<commit_before># Why did you choose Python for Machine Learning?

(Edit: This article is now also available in visually more appealing version [here](http://sebastianraschka.com/Articles/2015_why_python.html).)

Oh god, another one of those subjective, pointedly opinionated click-bait headlines? Yes! Why did I bother writing this? Well, here is one of the most trivial yet life-changing insights and worldly wisdoms from my former professor that has become my mantra ever since: "If you have to do this task more than 3 times just write a script and automate it."  

By now, you may have already started wondering about this blog. I haven't written anything for more than half a year! Okay, [musings on social network platforms](https://twitter.com/rasbt) aside, that's not true: I have written [something](https://github.com/rasbt/python-machine-learning-book) -- about 400 pages to be precise. This has really been quite a journey for me lately. And regarding the frequently asked question "Why did you choose Python for Machine Learning?" I guess it is about time to write  *my script*.


 In the following paragraphs, I really don't mean to tell you why *you* or anyone else should use Python. To be honest, I really hate those types of questions: "Which * is the best?" (* insert "programming language, text editor, IDE, operating system, computer manufacturer" here). This is really a nonsense question and discussion. Sometimes it can be fun and entertaining though, but I recommend saving this question for our occasional after-work beer or coffee with friends and colleagues.

<a class="mk-toclify" id="table-of-contents"></a>
#### Table of Contents

- [The short answer to a complex question](#the-short-answer-to-a-complex-question)
- [What are my favorite Python tools?](#what-are-my-favorite-python-tools)
- [What do I think about MATLAB?](#what-do-i-think-about-matlab)
- [Julia is awesome ... on paper!](#julia-is-awesome-on-paper)
- [There is really nothing wrong with R](#there-is-really-nothing-wrong-with-r)
- [What happened to Perl?](#what-happened-to-perl)
- [Other options](#other-options)
- [Is Python a dying language?](#is-python-a-dying-language)
- [Conclusion](#conclusion)
- [Feedback-and-opinions](#Feedback-and-opinions)



<a class="mk-toclify" id="the-short-answer-to-a-complex-question"></a>
## The short answer to a complex question
[[back to top](#table-of-contents)]


Maybe I should start with the short answer. You are welcome to stop reading this article below this paragraph because it really nails it. I am a scientist, I like to get my stuff done. I like to have an environment where I can quickly prototype and jot down my models and ideas. I need to solve very particular problems. I analyze given datasets to draw my conclusions. This is what matters most to me: How can I get the job done most productively? What do I mean by "productively"? Well, I typically run an analysis only once (the testing of different ideas and debugging aside); I don't need to repeatedly run a particular piece of code 24/7, I am not developing software applications or web apps for end users. When I *quantify* "productivity," I literally estimate the sum of (1) the time that it takes to get the idea written down in code, (2) debug it, and (3) execute it. To me, "most productively" means "how long does it take to get the results?" Now, over the years, I figured that Python is for me. Not always, but very often. Like everything else in life, Python is not a "silver bullet," it's not the "best" solution to every problem. However, it comes pretty close if you compare programming languages across the spectrum of common and not-so common problem tasks; Python is probably the most versatile and capable all-rounder.

![](./why-python/the_general_problem.png)
(Source: [https://xkcd.com/974/](https://xkcd.com/974/))

Remember: "Premature optimization is the root of all evil" (<NAME>). If you are part of the software engineering team that wants to optimize the next game-changing high-frequency trading model from your machine learning and data science division, Python is probably not for you (but maybe it was the language of choice by the data science team, so it may still be useful to learn how to read it). So, my little piece of advice is to evaluate your daily problem tasks and needs when you choose a language. "If all that you have is a hammer, everything starts to look like a nail" -- you are too smart to fall for this trap! However, keep in mind that there is a balance. There are occasions where the hammer may be the best choice even if a screwdriver would probably be the "nicer" solution. Again, it comes down to productivity.

**Let me give you an example from personal experience.**
I needed to develop a bunch of novel algorithms to "screen" 15 million small, chemical compounds with regard to a very problem specific hypothesis. I am an entirely computational person, but I am collaborating with biologists who do non-computational experiments (we call them "wet lab" experiments). The goal was to narrow it down to a list of 100 potential compounds that they could test in their lab. The caveat was that they needed the results quickly, because they only had limited time to conduct the experiments. 

> You’re bound to be unhappy if you optimize everything.         — <NAME>

Trust me, time was really "limited:" We just got our grant application accepted and research funded a few weeks before the results had to be collected (our collaborators were doing experiments on larvae of a certain fish species that only spawns in Spring). Therefore, I started thinking "How could I get those results to them as quickly as possible?" Well, I know C++ and FORTRAN, and if I implement those algorithms in the respective languages executing the "screening" run may be faster compared to a Python implementation. This was more of an educated guess, I don't really know if it would have been substantially faster. But there was one thing I knew for sure: If I started developing the code in Python, I could be able to get it to run in a few days -- maybe it would take a week to get the respective C++ versions coded up. I would worry about a more efficient implementation later. At that moment, it was just important to get those results to my collaborators -- "Premature optimization is the root of all evil." On a side node: The same train of thought applies to data storage solutions. Here, I just went with SQLite. CSV didn't make quite sense since I had to annotate and retrieve certain molecules repeatedly. I surely didn't want to scan or rewrite a CSV from start to end every time I wanted to look up a molecule or manipulate its entry -- issues in dealing with memory capacities aside. Maybe MySQL would have been even better but for the reasons mentioned above, I wanted to get the job done quickly, and setting up an additional SQL server ... there was no time for that, SQLite was just fine to get the job done.

![](./why-python/automation.png)

(Source: [https://xkcd.com/1319/](https://xkcd.com/1319/))

The verdict: **Choose the language that satisfies *your* needs!**
However, there is once little caveat here! How can a beginning programmer possibly know about the advantages and disadvantages of a language before learning it, and how should the programmer know if this language will be useful to her at all? This is what I would do: Just search for particular applications and solutions related to your most common problem tasks on Google and [GitHub](https://github.com). You don't need to read and understand the code. Just look at the end product.

> In the one and only true way. The object-oriented version of 'Spaghetti code' is, of course, 'Lasagna code'. (Too many layers). — <NAME>.

Also, don't hesitate to ask people. Don't just ask about the "best" programming language in general but be specific, describe your goals and why you want to learn how to program. If you want to develop applications for MacOS X you will probably want to check out Objective C and Swift, if you want to develop on Android, you are probably more interested in learning Java and so on.


<a class="mk-toclify" id="what-are-my-favorite-python-tools"></a>
## What are my favorite Python tools?
[[back to top](#table-of-contents)]


If you are interested, those are my favorite and most frequently used Python "tools," I use most of them on a daily basis.

- [NumPy](http://www.numpy.org): My favorite library for working with array structures and vectorizing equations using linear algebra; augmented by [SciPy](http://scipy2015.scipy.org/ehome/index.php?eventid=115969&).
- [Theano](https://theano.readthedocs.org/en/latest/): Implementing machine learning algorithms for the heavy-lifting and distributing computations across cores in my GPU(s).
- [scikit-learn](http://scikit-learn.org/stable/): The most convenient API for the daily, more basic machine learning tasks.
- [matplotlib](http://matplotlib.org): My library of choice when it comes to plotting. Sometimes I also use [seaborn](http://stanford.edu/~mwaskom/software/seaborn/index.html) for particular plots, for example, the heat maps are particularly great!
<commit_msg>fixed broken link to why-python post<commit_after># Why did you choose Python for Machine Learning?

(Edit: This article is now also available in visually more appealing version [here](http://sebastianraschka.com/blog/2015/why-python.html).)

Oh god, another one of those subjective, pointedly opinionated click-bait headlines? Yes! Why did I bother writing this? Well, here is one of the most trivial yet life-changing insights and worldly wisdoms from my former professor that has become my mantra ever since: "If you have to do this task more than 3 times just write a script and automate it."  

By now, you may have already started wondering about this blog. I haven't written anything for more than half a year! Okay, [musings on social network platforms](https://twitter.com/rasbt) aside, that's not true: I have written [something](https://github.com/rasbt/python-machine-learning-book) -- about 400 pages to be precise. This has really been quite a journey for me lately. And regarding the frequently asked question "Why did you choose Python for Machine Learning?" I guess it is about time to write  *my script*.


 In the following paragraphs, I really don't mean to tell you why *you* or anyone else should use Python. To be honest, I really hate those types of questions: "Which * is the best?" (* insert "programming language, text editor, IDE, operating system, computer manufacturer" here). This is really a nonsense question and discussion. Sometimes it can be fun and entertaining though, but I recommend saving this question for our occasional after-work beer or coffee with friends and colleagues.

<a class="mk-toclify" id="table-of-contents"></a>
#### Table of Contents

- [The short answer to a complex question](#the-short-answer-to-a-complex-question)
- [What are my favorite Python tools?](#what-are-my-favorite-python-tools)
- [What do I think about MATLAB?](#what-do-i-think-about-matlab)
- [Julia is awesome ... on paper!](#julia-is-awesome-on-paper)
- [There is really nothing wrong with R](#there-is-really-nothing-wrong-with-r)
- [What happened to Perl?](#what-happened-to-perl)
- [Other options](#other-options)
- [Is Python a dying language?](#is-python-a-dying-language)
- [Conclusion](#conclusion)
- [Feedback-and-opinions](#Feedback-and-opinions)



<a class="mk-toclify" id="the-short-answer-to-a-complex-question"></a>
## The short answer to a complex question
[[back to top](#table-of-contents)]


Maybe I should start with the short answer. You are welcome to stop reading this article below this paragraph because it really nails it. I am a scientist, I like to get my stuff done. I like to have an environment where I can quickly prototype and jot down my models and ideas. I need to solve very particular problems. I analyze given datasets to draw my conclusions. This is what matters most to me: How can I get the job done most productively? What do I mean by "productively"? Well, I typically run an analysis only once (the testing of different ideas and debugging aside); I don't need to repeatedly run a particular piece of code 24/7, I am not developing software applications or web apps for end users. When I *quantify* "productivity," I literally estimate the sum of (1) the time that it takes to get the idea written down in code, (2) debug it, and (3) execute it. To me, "most productively" means "how long does it take to get the results?" Now, over the years, I figured that Python is for me. Not always, but very often. Like everything else in life, Python is not a "silver bullet," it's not the "best" solution to every problem. However, it comes pretty close if you compare programming languages across the spectrum of common and not-so common problem tasks; Python is probably the most versatile and capable all-rounder.

![](./why-python/the_general_problem.png)
(Source: [https://xkcd.com/974/](https://xkcd.com/974/))

Remember: "Premature optimization is the root of all evil" (<NAME>). If you are part of the software engineering team that wants to optimize the next game-changing high-frequency trading model from your machine learning and data science division, Python is probably not for you (but maybe it was the language of choice by the data science team, so it may still be useful to learn how to read it). So, my little piece of advice is to evaluate your daily problem tasks and needs when you choose a language. "If all that you have is a hammer, everything starts to look like a nail" -- you are too smart to fall for this trap! However, keep in mind that there is a balance. There are occasions where the hammer may be the best choice even if a screwdriver would probably be the "nicer" solution. Again, it comes down to productivity.

**Let me give you an example from personal experience.**
I needed to develop a bunch of novel algorithms to "screen" 15 million small, chemical compounds with regard to a very problem specific hypothesis. I am an entirely computational person, but I am collaborating with biologists who do non-computational experiments (we call them "wet lab" experiments). The goal was to narrow it down to a list of 100 potential compounds that they could test in their lab. The caveat was that they needed the results quickly, because they only had limited time to conduct the experiments.

> You’re bound to be unhappy if you optimize everything.         — <NAME>

Trust me, time was really "limited:" We just got our grant application accepted and research funded a few weeks before the results had to be collected (our collaborators were doing experiments on larvae of a certain fish species that only spawns in Spring). Therefore, I started thinking "How could I get those results to them as quickly as possible?" Well, I know C++ and FORTRAN, and if I implement those algorithms in the respective languages executing the "screening" run may be faster compared to a Python implementation. This was more of an educated guess, I don't really know if it would have been substantially faster. But there was one thing I knew for sure: If I started developing the code in Python, I could be able to get it to run in a few days -- maybe it would take a week to get the respective C++ versions coded up. I would worry about a more efficient implementation later. At that moment, it was just important to get those results to my collaborators -- "Premature optimization is the root of all evil." On a side node: The same train of thought applies to data storage solutions. Here, I just went with SQLite. CSV didn't make quite sense since I had to annotate and retrieve certain molecules repeatedly. I surely didn't want to scan or rewrite a CSV from start to end every time I wanted to look up a molecule or manipulate its entry -- issues in dealing with memory capacities aside. Maybe MySQL would have been even better but for the reasons mentioned above, I wanted to get the job done quickly, and setting up an additional SQL server ... there was no time for that, SQLite was just fine to get the job done.

![](./why-python/automation.png)

(Source: [https://xkcd.com/1319/](https://xkcd.com/1319/))

The verdict: **Choose the language that satisfies *your* needs!**
However, there is once little caveat here! How can a beginning programmer possibly know about the advantages and disadvantages of a language before learning it, and how should the programmer know if this language will be useful to her at all? This is what I would do: Just search for particular applications and solutions related to your most common problem tasks on Google and [GitHub](https://github.com). You don't need to read and understand the code. Just look at the end product.

> In the one and only true way. The object-oriented version of 'Spaghetti code' is, of course, 'Lasagna code'. (Too many layers). — <NAME>.

Also, don't hesitate to ask people. Don't just ask about the "best" programming language in general but be specific, describe your goals and why you want to learn how to program. If you want to develop applications for MacOS X you will probably want to check out Objective C and Swift, if you want to develop on Android, you are probably more interested in learning Java and so on.


<a class="mk-toclify" id="what-are-my-favorite-python-tools"></a>
## What are my favorite Python tools?
[[back to top](#table-of-contents)]


If you are interested, those are my favorite and most frequently used Python "tools," I use most of them on a daily basis.

- [NumPy](http://www.numpy.org): My favorite library for working with array structures and vectorizing equations using linear algebra; augmented by [SciPy](http://scipy2015.scipy.org/ehome/index.php?eventid=115969&).
- [Theano](https://theano.readthedocs.org/en/latest/): Implementing machine learning algorithms for the heavy-lifting and distributing computations across cores in my GPU(s).
- [scikit-learn](http://scikit-learn.org/stable/): The most convenient API for the daily, more basic machine learning tasks.
- [matplotlib](http://matplotlib.org): My library of choice when it comes to plotting. Sometimes I also use [seaborn](http://stanford.edu/~mwaskom/software/seaborn/index.html) for particular plots, for example, the heat maps are particularly great!

{'fraction_non_alphanumeric': 0.0708229155342972, 'fraction_numerical': 0.004674421132731818, 'mean_word_length': 4.414655679811654, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 25, 'https://': 16, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '63564', 'n_tokens_mistral': 4758, 'n_tokens_neox': 4482, 'n_words': 2761}