<commit_before>---
layout: post
title: Notes on Time & Memory optimization with Python
---

In my Fanfic project I had to work with text data that adds up to ~36G (in CSV format). The server I use has 200G RAM, which seems big, but is actually not much when you manipulate data with this size. So, after repeatingly waiting for 3 days and getting a memory error, I realized I have to work on optimizing my code, which I had little training on. Therefore I'll put my notes on how to do it here, along with some take aways that I learned by the hard way.

I'm using Python and specifically the Pandas library to handle my data, so the notes will focus on how to save time/memory with them.




## Finding the bottle necks
Instead of randomly shooting, a smarter way is to first figure out which part of the code caused the slowliness/bulkiness. That is, we need a profile of the time/memory consumption for each part, ideally each line, of the code. 

For time profiling, Python comes with very good libraries [cProfile and Profile](https://docs.python.org/3.5/library/profile.html). cProfile is faster, so I went with it. 

It is easy to use the library:

`python -m cProfile -s cumtime unigram_turing_opt.py`

This is a screenshot of the output:

![image](yzjing.github.io/public/images/2017-03-25-time.png)

There are a few columns; `s` specifies which column to sort by. Here I sort by `cumtime`, so the code taking the most cumulative time is shown at the top. It is also easier to redirect the output to a file and `head` the file.

cProfile can also be used to time a single function. There is no need for me to describe the usage here.

We can also use external libraries. <NAME> has a [line_profiler](https://github.com/rkern/line_profiler) that allows line-to-line profiling. cProfile seems enough for me so I didn't really use it. It's well documented, so no more notes is needed here too.

For memory profiling, I used a library [memory_profler](https://pypi.python.org/pypi/memory_profiler) that is similar to the line_profiler. It can also profile the entire script or just one function. To profile my main() function, I can put a decorator before it:

	from memory_profiler import profile

	@profile
	def main(fandom):
    	print('working on fandom: ', fandom)
    	df = pd.read_csv(fandom + '_preprocessed.tsv', sep = '\t')

The output:

![image](yzjing.github.io/public/images/2017-03-25-memory.png)

I also read a nice blog post [How to profile memory usage in Python](https://www.pluralsight.com/blog/tutorials/how-to-profile-memory-usage-in-python) that explains how to use the library in details.

## Time optimizing notes
By doing the profiling described above and analyzing the reports, I was able to shorten the runtime of my code from 3 days to ~1.5 hours. Here are some main take aways:

1. NLTK's stemming is really slow. Stay away from it.
2. When working with Pandas dataframes, take out the unused columns ASAP.
3. Pandas's `apply` can be slow if you do it row by row (`axis=1`). `map` can be better, because `map` operates on the series and `apply` on the dataframe.

## Memory optimizing notes
I haven't totally sorted out my memory error, so will add this later.<commit_msg>update a post<commit_after>---
layout: post
title: Notes on Time & Memory optimization with Python
---

In my Fanfic project I had to work with text data that adds up to ~36G (in CSV format). The server I use has 200G RAM, which seems big, but is actually not much when you manipulate data with this size. So, after repeatingly waiting for 3 days and getting a memory error, I realized I have to work on optimizing my code, which I had little training on. Therefore I'll put my notes on how to do it here, so I don't have to Google it next time.





## Finding the bottle necks
Instead of randomly shooting, a smarter way is to first figure out which part of the code caused the slowliness/bulkiness. That is, we need a profile of the time/memory consumption for each part, ideally each line, of the code. 

For time profiling, Python comes with very good libraries [cProfile and Profile](https://docs.python.org/3.5/library/profile.html). cProfile is faster, so I went with it. 

It is easy to use the library:

`python -m cProfile -s cumtime unigram_turing_opt.py`

This is a screenshot of the output:

![image](yzjing.github.io/public/images/2017-03-25-time.png)

There are a few columns; `s` specifies which column to sort by. Here I sort by `cumtime`, so the code taking the most cumulative time is shown at the top. It is also easier to redirect the output to a file and `head` the file.

cProfile can also be used to time a single function. There is no need for me to describe the usage here.

We can also use external libraries. <NAME> has a [line_profiler](https://github.com/rkern/line_profiler) that allows line-to-line profiling. cProfile seems enough for me so I didn't really use it. It's well documented, so no more notes is needed here too.

For memory profiling, I used a library [memory_profler](https://pypi.python.org/pypi/memory_profiler) that is similar to the line_profiler. It can also profile the entire script or just one function. To profile my main() function, I can put a decorator before it:

	from memory_profiler import profile

	@profile
	def main(fandom):
    	print('working on fandom: ', fandom)
    	df = pd.read_csv(fandom + '_preprocessed.tsv', sep = '\t')

The output:

![image](yzjing.github.io/public/images/2017-03-25-memory.png)

I also read a nice blog post [How to profile memory usage in Python](https://www.pluralsight.com/blog/tutorials/how-to-profile-memory-usage-in-python) that explains how to use the library in details.

## Optimizing notes
By doing the profiling described above and analyzing the reports, I was able to shorten the runtime of my code from 3 days to ~1.5 hours. Here are some main take aways:

1. Don't use NLTK. Just don't.
2. When working with Pandas dataframes, take out the unused columns ASAP.
3. Pandas's `apply` can be slow if you do it row by row (`axis=1`). `map` can be better, because `map` operates on the series and `apply` on the dataframe.

{'fraction_non_alphanumeric': 0.07224396607958251, 'fraction_numerical': 0.010110893672537508, 'mean_word_length': 3.9821283509342, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1657666', 'n_tokens_mistral': 1824, 'n_tokens_neox': 1717, 'n_words': 941}