<jupyter_start><jupyter_text>Line-profilerThis tutorial covers the basics of the [line_profiler](https://github.com/rkern/line_profiler) Python package.  I'll cover what exactly line-profiler is (and why you should care) and how to use it. What is line-profilerAs the name suggests line-profiler profiles the runtime of a Python script in a line-by-line manner. From the profiling you can identify **hotspots**; i.e., what lines of code take the longest time to run! While sometimes this may be of simple academic interest, alternative algorithms could be used and lead to significant time savings.  If a loop is executed a million times and you save 10ms on each loop, congratulations your code now finishes three hours sooner!  Installing line-profilerProvided you have a working Python distribution, installing should be as easy as ```$ pip install line_profiler```If this fails, you can also clone the Github repository:```$ git clone https://github.com/rkern/line_profiler.git```And then install using ```$ python setup.py install```remembering you may need the ```--user``` or ```--prefix=/directory/to/local/python/site-packages/``` argument if you're on a cluster such as ```g2```. Using line-profiler with Jupyter NotebookWhen using line-profiler there is a slight twist depending on whether you're running the Python script directly from the terminal (e.g., ``python my_script.py``) or from inside a Jupyter notebook.  I'll go over both of these methods before showing some examples from my own code (which is all executed via command line).Loading line-profiler into your notebook is one easy command:<jupyter_code>%load_ext line_profiler<jupyter_output><empty_output><jupyter_text>Next we need some functions to test our run-time on.  For the first example, say we are given particle positions ``x``, ``y`` and ``z`` and we wish to calculate the pair-wise distance between each particle. The pair-wise distance is a very useful property which allows us to compute further statistics such as the correlation function or power spectrum, and is necessary for things such as force calculations.<jupyter_code>from __future__ import print_function # Always do this >:( 
from __future__ import division
import numpy as np

pos = np.random.random((1000,3)) # Generate a 3D vector of random points between 0 and 1.
print(pos[0])
print(pos[0,0])<jupyter_output>[ 0.39347777  0.82736576  0.15859656]
0.393477774111<jupyter_text>First we'll do my favourite method; the crass, brute force method that involves nested ``for`` loops.<jupyter_code>def pairwise_brute(pos):
    npart = pos.shape[0]
    ndim = pos.shape[1] 
    distance = np.empty((npart, npart), dtype = np.float64)
    
    # Note that the distance between pair i-j is the same as j-i.
    for i in range(npart): 
        for j in range(npart):
            d = 0.0
            for k in range(ndim):
                d += pow(pos[i,k] - pos[j,k], 2) # Gets the (square) distance in one dimension between particles i and j.
            distance[i,j] = np.sqrt(d)
        assert(distance[i,i] == 0) # A particle should have zero distance from itself.
    return distance

distance = pairwise_brute(pos)
print(distance[0,100]) # This is the distance between particle 0 and 100.<jupyter_output>0.212877826851<jupyter_text>Then we run ```line-profiler``` on our code.  The argument ``-f`` specifies the function that we want to profile and we need to remember to provide the input arguments.<jupyter_code>%lprun -f pairwise_brute pairwise_brute(pos)<jupyter_output><empty_output><jupyter_text>Timer unit: 1e-06 sTotal time: 9.24807 sFile: Function: pairwise_brute at line 1Line       Hits         Time  Per Hit   % Time  Line Contents==============================================================     1                                           def pairwise_brute(pos):     2         1            3      3.0      0.0      npart = pos.shape[0]     3         1            1      1.0      0.0      ndim = pos.shape[1]      4         1          677    677.0      0.0      distance = np.empty((npart, npart), dtype = np.float64)     5                                                    6                                                Note that the distance between pair i-j is the same as j-i.     7      1001          722      0.7      0.0      for i in range(npart):      8   1001000       480533      0.5      5.2          for j in range(npart):     9   1000000       415336      0.4      4.5              d = 0.0    10   4000000      2215483      0.6     24.0              for k in range(ndim):    11   3000000      4031247      1.3     43.6                  d += pow(pos[i,k] - pos[j,k], 2)  Gets the (square) distance in one dimension between particles i and j.    12   1000000      2102190      2.1     22.7              distance[i,j] = np.sqrt(d)    13      1000         1875      1.9      0.0          assert(distance[i,i] == 0)  A particle should have zero distance from itself.    14         1            1      1.0      0.0      return distance<jupyter_code>Be careful that in the above table the **Time** column is expressed in units of $10^{-6}s$, i.e., $\mu s$.  The **Time** column tells us how many $\mu s$ we spend on each line (total), the **Per Hit** column tells us the average amount of time spent on relative to the total amount of time spent in the function and finally the **% Time** column is the percentage of time spent on that line (relative to the total amount of time).

From this table a number of things become apparent.  Firstly, I enjoy writing long comments.  Secondly, one of the most expensive line of code is line 13 where we perform our ``assert`` error check.  This really highlights the power of a tool like line-profiler because whilst this line is expensive (taking $1.9\mu s$ per hit), the **relative** amount of time spent on this operation is small. The honour of most expensive operation goes to line 11 in which we spend $44\%$(!!) of our time on.  This is a result of our nested ``for`` loops; we are hitting line 11 a grand total of $4031247$ times.  

Let's see if we can do better.<jupyter_output><empty_output><jupyter_text>def pairwise_numpy(pos):    padded_pos = pos[:, None, :]  Pad the array to ensure the dimensionality will be correct     distance = np.sqrt(np.sum(np.square(padded_pos - pos), axis = -1))  Perform Pythag and then sum to return to the original dimensions.    return distance %lprun -f pairwise_numpy pairwise_numpy(pos)<jupyter_code>Timer unit: 1e-06 s

Total time: 0.050307 s
File: <ipython-input-77-fb2fce4afd83>
Function: pairwise_numpy at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           def pairwise_numpy(pos):
     2         1            8      8.0      0.0      padded_pos = pos[:, None, :] # Pad the array to ensure the dimensionality will be correct 
     3         1        50297  50297.0    100.0      distance = np.sqrt(np.sum(np.square(padded_pos - pos), axis = -1)) # Perform Pythag and then sum to return to the original dimensions.
     4         1            2      2.0      0.0      return distance<jupyter_output><empty_output><jupyter_text>Huzzah we have remarkable speed-up relative to the brute force method.  **Beware:** When using Numpy broadcasting, temporary arrays are created potentially causing memory issues for (very) large arrays.  Using line-profiler with terminalIf you're like me and behind the times, you can also run line-profiler from the terminal.  For this all you do is add the decorator ``@profile`` before the function you want to profile.  Then you profile the results by ```$ kernprof -l my_script.py ```If you get the error '``bash: kernprof: command not found``' then fully specify the path to where you installed ``line_profiler``:```$ python3 /home/jseiler/.local/lib/python3.5/site-packages/kernprof.py -l rewrite_files.py 15 15```Once the profiling has completed you can view the results via ```$ python3 -m line_profiler my_script.py.lprof```As an example from my own code (I've only included the lines with $>0.0$ in the **% Time** column):<jupyter_code>Timer unit: 1e-06 s

Total time: 2114.36 s
File: rewrite_files.py
Function: link_fof_snapshot_full at line 322

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   386       256      4468354  17454.5      0.2          with h5py.File(fname, 'r') as f:
   390       256      6629947  25898.2      0.3              BoxSize = f['Header'].attrs['BoxSize'] # Size of the simulation box (in Mpc/h).
   391       256        38943    152.1      0.0              Time = f['Header'].attrs['Time'] # Scale factor.
   392       256        37268    145.6      0.0              Omega_m = f['Header'].attrs['Omega0'] # Matter density.
   393       256        36343    142.0      0.0              Omega_l = f['Header'].attrs['OmegaLambda'] # Dark energy density.
   394       256        36325    141.9      0.0              particle_mass = f['Header'].attrs['MassTable'] # Table of particle masses (length equal to TypeMax)
   406       506      6359047  12567.3      0.3                              snapshot_partid = particles['ParticleIDs']
   407       506   1140906728 2254756.4     54.0                              ids_found = (np.nonzero(np.in1d(snapshot_partid, fof_partid[type_idx], assume_unique = True)))[0] # np.in1d returns a True if the snapshot particle is found in the FoF list (False otherwise).  np.nonzero then returns the indices of those values that have a 'True'.
   424       228     16848850  73898.5      0.8                                  particle_position[type_idx].append(particles['Coordinates'][ids_found]) # For some super weird reason this needs 3 indices to properly reference.  I.e. particle_position[1][0] will return all the coordinates for Particle Type 1.
   425       228     17104189  75018.4      0.8                                  particle_velocity[type_idx].append(particles['Velocities'][ids_found])
   426       228      1977059   8671.3      0.1                                  particle_id[type_idx].append(particles['ParticleIDs'][ids_found])
   429       228    915102216 4013606.2     43.3                                  fof_partid_idx = np.int32(np.nonzero(np.in1d(fof_partid[type_idx], snapshot_partid, assume_unique = True))[0])
   473       256      1306722   5104.4      0.1              print("Written Data to {0}.".format(fname))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10023685457129322, 'fraction_numerical': 0.06035054476551398, 'mean_word_length': 2.591697856413746, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '214244', 'license': 'MIT', 'n_tokens_mistral': 3330, 'n_tokens_neox': 2976, 'n_words': 1205, 'path': 'code-review/2017_12_07/line_profiler.py.ipynb', 'repo_name': 'aparthas3112/cookies-n-code', 'size': '10555'}