<commit_before>---
layout:     post
title:      One Program Written in Python, Go, and Rust
date:       2019-04-28
summary:    Image differentiation in three languages
categories: python go rust programming
published:  false
---

<!-- NOTE use textwidth=88 when wrapping -->

![Python, Go, Rust logos](/images/program-in-python-go-rust/python-go-rust.png)

_This is a subjective, primarily developer-ergonomics-based comparison of the
three languages from the perspective of a Python developer, but you can [skip to
performance comparison](#performance) if you want some hard numbers, [the
takeaway](#takeaway) if you just want a quick summary, or go straight to the
[Python](https://github.com/nicolashahn/diffimg),
[Go](https://github.com/nicolashahn/diffimg-go), and
[Rust](https://github.com/nicolashahn/diffimg-rs) implementations._

A few years ago, I was tasked with rewriting an image processing service so we could
host it in AWS Lambda. To tell whether my new service was creating the same output as
the old given an image and one or more transforms (resize to X by Y, make a circular
crop, etc.), I had to inspect the images myself. Clearly I needed to automate this, but
I could find no existing Python library that simply told me how different two images
were on a per-pixel basis.  Hence [diffimg](https://github.com/nicolashahn/diffimg),
which can give you a difference ratio/percentage, or generate a diff image (check out
the readme to see an example).

The initial implementation was in Python (the language I'm most comfortable in), with
the heavy lifting was done by
[Pillow](https://pillow.readthedocs.io/en/stable/). It's usable as a library or a
command line tool. The actual
[meat](https://github.com/nicolashahn/diffimg/blob/master/diffimg/diff.py) of the
program is very small, only a few dozen lines.  This is due to making use of Pillow's
`ImageChops.difference()` function, which generates the diff image - the ratio
calculation is only a few more lines. Not a lot of effort went into building this tool
([xkcd was right](https://xkcd.com/353/), there's a Python module for nearly
everything), but it's been very handy to many people.

A few months ago, I joined a company that had several services written in Go, and I need
to get up to speed quickly on the language. Writing
[diffimg-go](https://github.com/nicolashahn/diffimg-go) seemed like an fun and possibly
even useful way to do this. Here are a few points of interest that came out of the
experience, along with some that came up while using it at work: 

## Comparing Python and Go

(Again, the code: [diffimg](https://github.com/nicolashahn/diffimg) (python) and
[diffimg-go](https://github.com/nicolashahn/diffimg-go))

- __Standard Library__: Go comes with a decent [image](https://golang.org/pkg/image/)
  standard library module, as well as a command line argument parsing library. I didn't
  feel that I had to look for any external dependencies, and I ended up not needing them
  -- the `diffimg-go` implementation has none, where the Python implementation uses the
  fairly heavy third party module (ironically) named Pillow. I think Go's standard
  library in general is more structured and well thought out, Python's feels as if it
  were organically evolved, created by many authors over years, with many differing
  conventions. The Go standard library's consistency makes it easier to predict how any
  given module will function, and the source code is extremely well documented. 
  - One downside of using the standard image library is that it does not automatically
    detect if the image has an alpha channel, pixel values have four channels (RGBA) for
    all image types.  The `diffimg-go` implementation therefore requires the user to
    indicate whether or not they want to use the alpha channel for the calculation/image
    generation. This small inconvenience wasn't worth finding a third party library to
    fix, in my opinion.
  - One big upside is that there's enough in the standard library that you don't need a
    web framework like Django. You can build out a real, usable API without any
    dependencies. Python's claim in the past has been that it's batteries-included, but
    Go does it better, in my opinion.

- __Static Type System__: Using one was somewhat foreign to me, as almost all
  my programming for the past few years has been in Python. The experience was somewhat
  annoying at first, it felt as though it was simply slowing me down and forcing me to
  be excessively explicit whereas Python would just let me do what I wanted, even if I
  got it wrong occasionally.  Somewhat like giving instructions to someone who always
  stops you to ask you to clarify what you mean, versus someone who always nods along
  and seems to understand you, though you're not always sure they're absorbing
  everything. It can help decrease a certain class of bugs but the presence of a static
  type system does not mean you don't need to write tests.
  - One of the common complaints of Go is that it does not have user-implementable
    generic types. While this is not a must-have feature for building a large,
    extensible application, it certainly slows development speed.
    [Alternatives](https://appliedgo.net/generics/) have been suggested, but none of
    them are as satisfying or effective as having real generic types.

- __Verbosity__: Go is much more verbose (though not Java verbose). Part of that is
  because type system does not have generics, but mainly the fact that the language
  itself is very small and not overloaded with features (you only get [one looping
  construct!](https://tour.golang.org/flowcontrol/1) I missed having list comprehensions
  and other functional programming features), not that Python is. If you're comfortable
  with Python, you can go through the [Tour of Go](https://tour.golang.org/welcome/1) in
  a day or two, and you'll have been exposed to the entirety of the language. How I felt
  at the end of it:

  ![travolta_meme.gif](/images/travolta_meme.gif?style=centered)

- __Interfaces and Structs__: Go uses interfaces and structs where Python would use
  classes (you can simulate interfaces in Python by writing classes that only have
  virtual methods, though I'm not sure if this is useful). This was probably the most
  interesting difference to me, as it forced me to differentiate the concept of a type
  that defines behavior versus a type that holds information.  Python and any other OO
  language would encourage you to mash these together, but there are pros and cons to
  both paradigms:
  - Divorcing implementations for interfaces means you need to write similar code
    several times if you have many types that are similar to each other. This is seems
    to me a "bottom-up" approach to writing abstractions, whereas Python would be
    "top-down". This style is still new to me, so I'm still undecided as to whether this
    is a pattern I like - it does seem to require more code, which I'm initially put off
    by.
  - However, because Go is statically typed, the compiler/linter will tell you when
    you're writing code that would have caused a runtime error in Python when you try to
    access a method or attribute that may not exist. Forcing you to write
    implementations for specific types is a requirement for this. You can be much more
    sure of your Go code when it compiles. There are more chances to make mistakes, but
    more help in catching them. Python linters can get some of this functionality, but
    because of the language's dynamicity, it's not foolproof like Go is.

- __Error Handling__: Python has exceptions, whereas Go propagates errors by returning
  tuples: `value, error` from functions wherever something may go wrong. Python lets
  you catch errors at any point in the call stack as opposed to requiring you to
  manually pass them back up over and over again. This again results in brevity and
  code that isn't littered with `if err != nil`s, though you do need to be aware of
  what possible exceptions can be thrown by a function and all(!) of its internal
  calls (using `except Exception:` is a bad-practice workaround for this). Good
  docstrings and tests can help here, which you should be writing in either language.
  Go's system is definitely safer. You're still allowed to shoot yourself in the foot
  by just ignoring the `err` second tuple value, but it's more obvious that this is a
  bad idea.

- __Optional Arguments__: Go has none, Python makes extensive use of them. A seemingly
  small feature, but found it was something I really missed, mainly for how much
  easier refactoring is if you can just throw a `kwarg` onto whatever function needs
  it without having to rewrite every one of its calls. This made my implementation for
  how to handle whether or not the diff image should be created based on the command
  line flags somewhat clumsy.

- __Third Party Modules__: Prior to [Go modules](https://blog.golang.org/modules2019),
  Go's package manager would just throw all downloaded packages into `$GOPATH/src`
  instead of the project's directory like every other language. The path for these
  modules inside `$GOPATH` would also be built from the URL, so your import would look
  something like `import "github.com/someuser/somepackage"`. Embedding `github.com`
  inside the source code of almost all Go codebases seems like a strange choice. In any
  case, Go has come around to the conventional way of doing things, but Go modules are<commit_msg>more on interfaces and structs<commit_after>---
layout:     post
title:      One Program Written in Python, Go, and Rust
date:       2019-04-28
summary:    Image differentiation in three languages
categories: python go rust programming
published:  false
---

<!-- NOTE use textwidth=88 when wrapping -->

![Python, Go, Rust logos](/images/program-in-python-go-rust/python-go-rust.png)

_This is a subjective, primarily developer-ergonomics-based comparison of the
three languages from the perspective of a Python developer, but you can [skip to
performance comparison](#performance) if you want some hard numbers, [the
takeaway](#takeaway) if you just want a quick summary, or go straight to the
[Python](https://github.com/nicolashahn/diffimg),
[Go](https://github.com/nicolashahn/diffimg-go), and
[Rust](https://github.com/nicolashahn/diffimg-rs) implementations._

A few years ago, I was tasked with rewriting an image processing service so we could
host it in AWS Lambda. To tell whether my new service was creating the same output as
the old given an image and one or more transforms (resize to X by Y, make a circular
crop, etc.), I had to inspect the images myself. Clearly I needed to automate this, but
I could find no existing Python library that simply told me how different two images
were on a per-pixel basis.  Hence [diffimg](https://github.com/nicolashahn/diffimg),
which can give you a difference ratio/percentage, or generate a diff image (check out
the readme to see an example).

The initial implementation was in Python (the language I'm most comfortable in), with
the heavy lifting was done by
[Pillow](https://pillow.readthedocs.io/en/stable/). It's usable as a library or a
command line tool. The actual
[meat](https://github.com/nicolashahn/diffimg/blob/master/diffimg/diff.py) of the
program is very small, only a few dozen lines.  This is due to making use of Pillow's
`ImageChops.difference()` function, which generates the diff image - the ratio
calculation is only a few more lines. Not a lot of effort went into building this tool
([xkcd was right](https://xkcd.com/353/), there's a Python module for nearly
everything), but it's been very handy to many people.

A few months ago, I joined a company that had several services written in Go, and I need
to get up to speed quickly on the language. Writing
[diffimg-go](https://github.com/nicolashahn/diffimg-go) seemed like an fun and possibly
even useful way to do this. Here are a few points of interest that came out of the
experience, along with some that came up while using it at work: 

## Comparing Python and Go

(Again, the code: [diffimg](https://github.com/nicolashahn/diffimg) (python) and
[diffimg-go](https://github.com/nicolashahn/diffimg-go))

- __Standard Library__: Go comes with a decent [image](https://golang.org/pkg/image/)
  standard library module, as well as a command line argument parsing library. I didn't
  feel that I had to look for any external dependencies, and I ended up not needing them
  -- the `diffimg-go` implementation has none, where the Python implementation uses the
  fairly heavy third party module (ironically) named Pillow. I think Go's standard
  library in general is more structured and well thought out, Python's feels as if it
  were organically evolved, created by many authors over years, with many differing
  conventions. The Go standard library's consistency makes it easier to predict how any
  given module will function, and the source code is extremely well documented. 
  - One downside of using the standard image library is that it does not automatically
    detect if the image has an alpha channel, pixel values have four channels (RGBA) for
    all image types.  The `diffimg-go` implementation therefore requires the user to
    indicate whether or not they want to use the alpha channel for the calculation/image
    generation. This small inconvenience wasn't worth finding a third party library to
    fix, in my opinion.
  - One big upside is that there's enough in the standard library that you don't need a
    web framework like Django. You can build out a real, usable API without any
    dependencies. Python's claim in the past has been that it's batteries-included, but
    Go does it better, in my opinion.

- __Static Type System__: Using one was somewhat foreign to me, as almost all
  my programming for the past few years has been in Python. The experience was somewhat
  annoying at first, it felt as though it was simply slowing me down and forcing me to
  be excessively explicit whereas Python would just let me do what I wanted, even if I
  got it wrong occasionally.  Somewhat like giving instructions to someone who always
  stops you to ask you to clarify what you mean, versus someone who always nods along
  and seems to understand you, though you're not always sure they're absorbing
  everything. It can help decrease a certain class of bugs but the presence of a static
  type system does not mean you don't need to write tests.
  - One of the common complaints of Go is that it does not have user-implementable
    generic types. While this is not a must-have feature for building a large,
    extensible application, it certainly slows development speed.
    [Alternatives](https://appliedgo.net/generics/) have been suggested, but none of
    them are as satisfying or effective as having real generic types.

- __Verbosity__: Go is much more verbose (though not Java verbose). Part of that is
  because type system does not have generics, but mainly the fact that the language
  itself is very small and not overloaded with features (you only get [one looping
  construct!](https://tour.golang.org/flowcontrol/1) I missed having list comprehensions
  and other functional programming features), not that Python is. If you're comfortable
  with Python, you can go through the [Tour of Go](https://tour.golang.org/welcome/1) in
  a day or two, and you'll have been exposed to the entirety of the language. How I felt
  at the end of it:

  ![travolta_meme.gif](/images/travolta_meme.gif?style=centered)

- __Interfaces and Structs__: Go uses interfaces and structs where Python would use
  classes (you can simulate interfaces in Python by writing classes that only have
  virtual methods, though I'm not sure if this is useful). This was probably the most
  interesting difference to me, as it forced me to differentiate the concept of a type
  that defines behavior versus a type that holds information.  Python and any other OO
  language would encourage you to mash these together, but there are pros and cons to
  both paradigms:
  - Go heavily encourages [composition over
    inheritance](https://en.wikipedia.org/wiki/Composition_over_inheritance). While it
    has [inheritance via embedding](https://golang.org/doc/effective_go.html#embedding),
    without classes, it's not easy to forward both data and methods. I generally agree
    that composition is the better default pattern to reach for, but I'm not an
    absolutist and some solutions are a better fit for inheritance, so I'd prefer not to
    have the language make this decision for me. I assume once you're more comfortable
    with the pattern, it's not an issue anymore, and coming up with composition-only
    code is second nature.
  - Divorcing implementations for interfaces means you need to write similar code
    several times if you have many types that are similar to each other. Because of the
    lack of generic types, there are situations in Go where I wouldn't be able to reuse
    code, though I would in Python.
  - However, because Go is statically typed, the compiler/linter will tell you when
    you're writing code that would have caused a runtime error in Python when you try to
    access a method or attribute that may not exist. Forcing you to write
    implementations for specific types is a requirement for this. Go that compiles is
    more trustworthy than Python that runs. Python linters can get some of this
    functionality, but because of the language's dynamicity, it's not nearly as good as
    Go's.

- __Error Handling__: Python has exceptions, whereas Go propagates errors by returning
  tuples: `value, error` from functions wherever something may go wrong. Python lets
  you catch errors at any point in the call stack as opposed to requiring you to
  manually pass them back up over and over again. This again results in brevity and
  code that isn't littered with `if err != nil`s, though you do need to be aware of
  what possible exceptions can be thrown by a function and all(!) of its internal
  calls (using `except Exception:` is a bad-practice workaround for this). Good
  docstrings and tests can help here, which you should be writing in either language.
  Go's system is definitely safer. You're still allowed to shoot yourself in the foot
  by just ignoring the `err` second tuple value, but it's more obvious that this is a
  bad idea.

- __Optional Arguments__: Go has none, Python makes extensive use of them. A seemingly
  small feature, but found it was something I really missed, mainly for how much
  easier refactoring is if you can just throw a `kwarg` onto whatever function needs
  it without having to rewrite every one of its calls. This made my implementation for
  how to handle whether or not the diff image should be created based on the command
  line flags somewhat clumsy.

- __Third Party Modules__: Prior to [Go modules](https://blog.golang.org/modules2019),
  Go's package manager would just throw all downloaded packages into `$GOPATH/src`
  instead of the project's directory like every other language. The path for these
  modules inside `$GOPATH` would also be built from the URL, so your import would look
  something like `import "github.com/someuser/somepackage"`. Embedding `github.com`
  inside the source code of almost all Go codebases seems like a strange choice. In any
  case, Go has come around to the conventional way of doing things, but Go modules are
{'fraction_non_alphanumeric': 0.05279057859703021, 'fraction_numerical': 0.0019457245263696877, 'mean_word_length': 4.009233136701718, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 32, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6538363', 'n_tokens_mistral': 5087, 'n_tokens_neox': 4854, 'n_words': 2912}