<commit_before>---
layout: post
title:  Julia in safe Data Science
date:   2016-04-18
tag: data-science
categories:
  - data-science
  - julia
keywords: "Julia, Data Science"
author: cantuna
---

<!--excerpt.start-->
Recently in Jampp, I had the chance to switch some of our data science environment from Python to Julia. For various reasons, it's type system is, in my opinion, one of the best language features. The most obvious reason are performance enhancements it allows. I will not, however, address that point here: it has been benchmarked very well in several places. Instead, I will briefly show a safety advantage this type system brings that is really handy for data science.
<!--excerpt.end-->

To be fair, this feature is handy in a production oriented data science team. Someone more oriented towards analytics, then, might not find it so helpful. Quoting [<NAME>][CHANG], I perform Type B data science: models I build are meant to get into production. Because of this fact, safety issues are a big concern when implementing them. If we don't catch a misbehaviour in our Machine Learning systems, we will be running our business based on mistaken metrics. For example, bidding high prices for bad banners millions times a day...

So, let's get to the point. When you deal with hundreds of thousands of messages per second coming from a number of sources, you learn one thing: every field will eventually receive at least one ill-formated message. It is true, and not only for data science, that a smart use of types in a language helps you finding mismatches early in the data handling process. Take this toy example and look when Python realizes there is something wrong with the message and when Julia does:

Here is Python's version:
{% highlight python %}
import pandas as pd
import numpy as np

#I received a few messages
df = pd.DataFrame(np.round(np.random.rand(1000, 200)),
          dtype=float,
          columns = ["Attribute{}".format(d) for d in range(1,201)])<commit_msg>Small fixes in English style.<commit_after>---
layout: post
title:  Julia in safe Data Science
date:   2016-04-18
tag: data-science
categories:
  - data-science
  - julia
keywords: "Julia, Data Science"
author: cantuna
---

<!--excerpt.start-->
Recently in Jampp, I had the chance to switch some of our data science environment from Python to Julia. For various reasons, its type system is, in my opinion, one of the best language features. The most obvious one is the performance enhancements it allows. I will not, however, address that point here: it has been benchmarked very well in several places. Instead, I will briefly show a safety advantage this type system brings that is really handy for data science.
<!--excerpt.end-->

To be fair, this feature is handy in a production oriented data science team. Someone more oriented towards analytics, then, might not find it so helpful. Quoting [<NAME>][CHANG], I perform Type B data science: models I build are meant to get into production. Because of this fact, safety issues are a big concern when implementing them. If we don't catch a misbehaviour in our Machine Learning systems, we will be running our business based on mistaken metrics. For example, bidding high prices for bad banners millions times a day...

So, let's get to the point. When you deal with hundreds of thousands of messages per second coming from a number of sources, you learn one thing: every field will eventually receive at least one ill-formated message. It is true, and not only for data science, that a smart use of types in a language helps you finding mismatches early in the data handling process. Take this toy example and look when Python realizes there is something wrong with the message and what Julia does:

Here is Python's version:
{% highlight python %}
import pandas as pd
import numpy as np

#I received a few messages
df = pd.DataFrame(np.round(np.random.rand(1000, 200)),
          dtype=float,
          columns = ["Attribute{}".format(d) for d in range(1,201)])
{'fraction_non_alphanumeric': 0.061768368617683686, 'fraction_numerical': 0.009464508094645082, 'mean_word_length': 4.115923566878981, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6813518', 'n_tokens_mistral': 1027, 'n_tokens_neox': 962, 'n_words': 618}