<commit_before>---
layout: post
title:  "Building an analysis monitor for a genomics pipeline: Tracking remote analyses"
date: 2015-11-11
comments: true
tags:
    - bioinformatics
    - SSE
    - Data analysis
    - Software architecture
---

![monitor]({{ site.url }}/assets/images/bcbio-monitor/logo-letters.png)

A typical genomics data analysis lasts usually for several hours or even days, depending on the type
of analysis performed. A common problem with these kind of analysis is that sometimes its hard, or
inconvenient at least, to know the current status of the analysis being performed. In most cases
it involves SSH-ing to remote machines, looking for logs and GREP-ing through them.

[bcbio-nextgen][bcbio] is a python toolkit providing best-practice pipelines for fully automated high
throughput sequencing analysis.

<!--more-->

# The problem
As mentioned before, tracking long-run analyses is not always easy on these kind of workflows, and bcbio-nextgen
is not an exception. When running an analysis, the only way to know what's the status of it is by looking
into the logs. Wouldn't it be nice, and more convenient, to have a simple frontend layer that displayed
the progress of the current analysis? What if it could do so independently from the pipeline itself?

# The solution
That's precisely what [bcbio-monitor][monitor] does. Now, to build a frontend for such a piece of software
like **bcbio-nextgen**, there are a couple of architectural **decisions** one needs to take. The very first
question that came to my mind was:

## Coupled or decoupled?
Do you want the tracker to be tighten to the software its tracking, in the sense of _requiring_ it (installed) to
function? You may think that's the logical choice at first, but think about this: bcbio-nextgen, as many
complex analysis pipelines has a lot of dependencies and its tricky to install. Even with [all the facilities][installation]
that [Brad Chapman][brad] gives us to install bcbio, still time and computational resources are needed
to install bcbio-nextgen. I thought that, to be a practical tool, the potential users should be able to
install the monitor real quick _on a personal computer_ and start tracking analyses wherever they're running
(which is generally a remote HPC or Cloud environment).

Every programmer should know the [importance of good logging practices][logging] for any application, right? bcbio-nextgen
does this _really_ well. So I though that I could just use the information from the log that bcbio-nextgen
generates, instead of inserting tracking code in the pipeline itself. **This makes the monitor independent
from the pipeline**, making it possible to just read ongoing analyses log files to build a status graph,
or even read finished analyses logs to get a global summary.

## Updating the client (web frontend)
Having decided to use the log file, we need a way to update the client application on every pipeline
step. There are mainly two options:
<commit_msg>Apply suggestions by @adcaes<commit_after>---
layout: post
title:  "Building an analysis monitor for a genomics pipeline: Tracking remote analyses"
date: 2015-11-11
comments: true
tags:
    - bioinformatics
    - SSE
    - Data analysis
    - Software architecture
---

![monitor]({{ site.url }}/assets/images/bcbio-monitor/logo-letters.png)

A typical genomics data analysis lasts usually for several hours or even days, depending on the type
of analysis performed. A common problem with these kind of analysis is that sometimes its hard, or
inconvenient at least, to know the current status of the analysis being performed. In most cases
it involves SSH-ing to remote machines, looking for logs and GREP-ing through them.

[bcbio-nextgen][bcbio] is a python toolkit providing best-practice pipelines for fully automated high
throughput sequencing analysis.

<!--more-->

# The problem
As mentioned before, tracking long-run analyses is not always easy on these kind of workflows, and bcbio-nextgen
is not an exception. When running an analysis, the only way to know what's the status of it is by looking
into the logs. Wouldn't it be nice, and more convenient, to have a simple frontend layer that displayed
the progress of the current analysis? What if it could do so independently from the pipeline itself?

# The solution
That's precisely what [bcbio-monitor][monitor] does. Now, to build a frontend for such a piece of software
like **bcbio-nextgen**, there are a couple of architectural **decisions** one needs to take. The very first
question that came to my mind was:

## Coupled or decoupled?
Do you want the tracker to be tighten to the software its tracking, in the sense of _requiring_ it (installed) to
function? You may think that's the logical choice at first, but think about this: bcbio-nextgen, as many
complex analysis pipelines has a lot of dependencies and its tricky to install. Even with [all the facilities][installation]
that [<NAME>][brad] gives us to install bcbio, still time and computational resources are needed
to install bcbio-nextgen. I thought that, to be a practical tool, the potential users should be able to
install the monitor really quick _on a personal computer_ and start tracking analyses wherever they're running
(which is generally a remote HPC or Cloud environment).

Every programmer should know the [importance of good logging practices][logging] for any application, right? bcbio-nextgen
does this _really_ well. So I though that I could just use the information from the log that bcbio-nextgen
generates, instead of inserting tracking code in the pipeline itself. **This makes the monitor independent
from the pipeline**, making it possible to just read ongoing analyses log files to build a status graph,
or even read finished analyses logs to get a global summary.

## Updating the web client
Having decided to use the log file, we need a way to update the client application on every pipeline
step. There are mainly two options:

{'fraction_non_alphanumeric': 0.05144101876675603, 'fraction_numerical': 0.002680965147453083, 'mean_word_length': 4.568097014925373, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3203150', 'n_tokens_mistral': 1532, 'n_tokens_neox': 1446, 'n_words': 899}