<commit_before>---
layout:     post
title:      "Why we moved from Heroku to EC2"
subtitle:   ""
description: "Why we went from a 100% PaaS infrastructure to an hybrid solution with a mix of Heroku for our API and EC2 for job processing"
date:       2014-10-21
author:     "<NAME>"
header-img: "img/post-bg-06.jpg"
---


###Our context

At [Youboox](http://www.youboox.fr) we process a lot of books, when we receive a new book we start a backround job that will :

* Compress the book (ex : extracting image, compressing them, etc)
* Slice the book into several smaller files, so we can send only pages corresponding to the current position of the user in the book
* Crypt the file to apply our custom DRM

These jobs take between a few minutes to almost an hour for the larger books (some of the files we receive are over 100 Mo)

Over the time we discovered the challenges of processing long and complex background jobs on Heroku, we'll see how we solved them by moving our book pour processing infrastructure to EC2.



### Why we started with Heroku :

Late 2011 when we started to write the first lines of code for Youboox, we had very limited ressources and a lot of things to do. The beta service included :

* 1 Rails app for publisher to upload their books
* 1 iPad app
* 1 json API in rails

At that time we had 2 priorities : signing deal with publishers and growing our user base.

Our technical choices aimed at delivering value as fast as possible to users and publishers. We were focused on development rather than managing infrastructure, so Heroku was a perfect fit in this situation, it was great to get up and running fast without ever having to ssh to a server.
If i had to do it again i would definitely go for Heroku.


### Our book processing infrastructure :

At the betinning of 2012 after a few months of development we released our beta, this is how our  background job infrastructure looked like :

<img src="/img//posts/2014-12-23-Why-we-moved-from-Heroku-to-EC2/architecture-2012.jpg" width="800">

It was fairly simple :
A publisher would upload a book on our web app
It started a background job, that includes the following steps :

* Compressing
* Slicing
* crypting

When the job was done the files were stored on AWS S3


### Monitoring problems :

This simple approach roughly worked. But we soon discovered it had several drawbacks :<br>
The book industry is very different from the music or film industry, the standard for digital books is the epub format, problem is every publisher is doing it their own way.<br>
Everytime we would sign a new publisher we received a different epub format...
<commit_msg>improve intro<commit_after>---
layout:     post
title:      "Why we moved from Heroku to EC2"
subtitle:   ""
description: "Why we went from a 100% PaaS infrastructure to an hybrid solution with a mix of Heroku for our API and EC2 for job processing"
date:       2014-10-21
author:     "<NAME>"
header-img: "img/post-bg-06.jpg"
---

### Introduction

[Youboox](http://www.youboox.fr) is a young startup offering a subscription to ebooks. Over the time we discovered the challenges of processing long and complex background jobs on Heroku, we'll see how we solved them by moving our book pour processing infrastructure to EC2.


### Why we started with Heroku :

Late 2011 when we started to write the first lines of code for Youboox, we had very limited ressources and a lot of things to do. The beta service included :

* 1 Rails app for publisher to upload their books
* 1 iPad app
* 1 json API in rails

At that time we had 2 priorities : signing deal with publishers and growing our user base.

Our technical choices aimed at delivering value as fast as possible to users and publishers. We were focused on development rather than managing infrastructure, so Heroku was a perfect fit in this situation, it was great to get up and running fast without ever having to ssh to a server. If i had to do it again i would definitely go for Heroku.


###Our context

When we receive a new book we start a backround job that will :

* Compress the book (ex : extracting image, compressing them, etc)
* Slice the book into several smaller files, so we can send only pages corresponding to the current position of the user in the book
* Crypt the file to apply our custom DRM

These jobs take between a few minutes to almost an hour for the larger books (some of the files we receive are over 100 Mo)


### Our book processing infrastructure :

At the betinning of 2012 after a few months of development we released our beta, this is how our  background job infrastructure looked like :

<img src="/img//posts/2014-12-23-Why-we-moved-from-Heroku-to-EC2/architecture-2012.jpg" width="800">

It was fairly simple :
A publisher would upload a book on our web app
It started a background job, that includes the following steps :

* Compressing
* Slicing
* crypting

When the job was done the files were stored on AWS S3


### Monitoring problems :

This simple approach roughly worked. But we soon discovered it had several drawbacks :<br>
The book industry is very different from the music or film industry, the standard for digital books is the epub format, problem is every publisher is doing it their own way.<br>
Everytime we would sign a new publisher we received a different epub format...

{'fraction_non_alphanumeric': 0.050478513792456374, 'fraction_numerical': 0.018014636892475136, 'mean_word_length': 3.9215143120960296, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6813514', 'n_tokens_mistral': 1441, 'n_tokens_neox': 1335, 'n_words': 854}