Category Archives: Taverna
2008-04-16 Stian Soiland-Reyes	1 Comment
I'm currently looking into an old bug, TAV-480, that seems to have reappeared. This bug is related to Raven, our classloader system based on Maven that allows us to do plugins and network updates. I'll come with another post about this bug, but first let's do an introduction of Raven.
The idea of Raven is quite simple, we use Maven to deploy all our modules, and all of our dependencies are also available either from our own or from official third-party Maven repositories. One thing we decided to support from Taverna 1.5 was plugins and dynamic updates over the web. How could we do this?
We decided to reuse the existing Maven infrastructure also at runtime. To install a plugin would then simply be a matter of specifying which Maven artifacts are needed, and from which Maven repositories to get the artifacts, such as in the plugin description for the t2 preview plugin. So we built Raven, which although inspired by Maven doesn't use any Maven code.
Given a Maven artifact as described in the plugin file:
<artifact groupId="net.sf.taverna.t2"
artifactId="biomart-activity" version="0.2.0"/>
Raven fetches the POM and JAR files for biomart-activity from the repository, parses the Maven artifact description file biomart-activity-0.2.0.pom, and then does the same for each of the dependencies listed, and with their dependencies again, and so on until everything required has been downloaded.
What remains is to get all these JARs available on the classpath. The normal way to do this is to use a ClassLoader, such as the URLClassLoader. You can't normally modify the classloader Java gives you at startup, because it has been determined from the -classpath parameter and you can't officially add new URLs to it once it's constructed. However you are free to construct a new classloader and load classes from there.
This is what the bootstrapper asks Raven to do. Since we can have several plugins, each plugin would need its own classloader. The plugins can come from different third-parties (you can add plugin sites to Taverna), and they might depend on different versions of common artifacts such as jdom. In some cases these versions are not compatible.
What we ended up with as a solution was that each Maven artifact gets its own LocalArtifactClassLoader. This classloader also have a list of the dependencies as declared in the POM, and when asked to resolve a classname, it will first search each of it's dependencies. As the dependencies have the same kind of classloaders, they will use the same logic, and the search will be depth-first. If none of the dependencies have the class, then maybe the JAR file associated with this artifact does, so it does a normal search through its own superclass URLClassLoader which searches the single JAR file. If this fails, a ClassNotFoundException is thrown and whoever depended on this artefact would go on to check the next dependency.
When you download Taverna these days, what you get in the zip-file is a tiny bootstrapper class together with a configuration file raven.properties that says which are the global Maven repositories, and which profile should be used for the main program. The profile version 1.7.1.0 basically says what is Taverna 1.7.1.0 by listing all the required artifacts, very similar to the plugin definition. For instance you would find all the different processors listed here.
If we decide that we need to publish an update of a certain processor, we can just deploy the new version to the Maven repositories, and publish a new version of this profile and list it in the profile list that Taverna checks on startup.
Although this solution came with a few quirks we had to work out (for instance many of the official POMs didn't correctly state what where the true dependencies of the library), in general it's a nice solution with lots of possibilities, for instance it doesn't prevent you to have two versions of the same artifact at once, so you could theoretically build a workflow that used both an old and a new version of the wsdl-activity – if for instance we had fixed a bug that you suspect might have affected some old workflow runs and you want to compare the outputs.
However, one of the difficulties with doing all of the ClassLoader work ourself is that as this is quite low-level and hard-core Java with many tiny little pitfalls to worry about and lots of concepts to understand. I'll come back in the next post with the example related to our bug TAV-480.
If you are interested in using Raven in your own project, contact us on taverna-hackers!
classloadermavenpluginsraven
Releases, Taverna
Taverna 1.7.1 released
Taverna 1.7.1 was released today, including an updated preview of t2, the enactor of Taverna 2.0 scheduled for June 2008.
You can download 1.7.1 from SourceForge.
jirareleaset2Taverna
Stand up and be counted
Taverna workflows are full of shims. That's a fact. Shims are the little adapter services, mostly using Beanshells, which convert the outputs of one workflow processor before sending it to the input of another. They are always being re-invented and 90% of the time do the same things – concatentate strings, swap things around…..
The problem is that these shims are designed to be use once, almost throw away, so they are not annotated with any sort of helpful information. This becomes a problem with the scientists greatest challenge – provenance and data lineage (OK, I might be being a bit melodramatic about it being the greatest challenge but it is up there somewhere – maybe nearer to the challenge of making the perfect cup of tea, no mean feat). Data goes into a shim and comes out the other side but what is happening inside? Shims are not the only processors guilty of this, there are plenty of black box services out there in the world. So, how can we address this problem? Well, we have started to collect the shims that people actually use in a my experiment group, we will then figure out all the similarities and come up with an annotated set which we can all use from Taverna 2 (T2). The current idea is for the T2 workbench to have an intelligent workflow designer which will recognise that you are trying to do some shim magic and suggest one to use. Maybe we will need a Taverna clippy style pop up (think Word etc) – ideas on a postcard……
So, if you are a shim it's time to stand up and announce to the world – "I'm a shim and I'm proud of it".
data lineageprovenanceshimst2Tavernaworkflow
Academic, Taverna
Data referencing in action in "Integrating ARC Grid with Taverna"
2008-03-31 Stian Soiland-Reyes	2 Comments
Daniel Bayer at University of Lübeck has implemented an ARC Grid plugin for Taverna for submitting job to the grid system ARC, in particular for usage with KnowARC. Bayer, together with Steffen Möller and Hajo N. Krabbenhöft also recently got published in Bioinformatics.
The work seems very promising, and in particular because it seems to have much of the same inspiration as t2 with regards to security and referenced data, although their plugin is for Taverna 1.6.2, before even the t2 plugin was included in Taverna.
arcgridpluginsecurityTaverna
Code freeze of Taverna 1.7.1
2008-03-28 Stian Soiland-Reyes	Leave a comment
We're hopefully doing our code freeze for Taverna 1.7.1 today, 2008-03-28.
There's a few bug fixes, in addition to quite a bit of work on t2. There will also be a drag-and-drop-able, interactive workflow editor, basically you can edit the workflow by connecting lines between the processors. Those of you who remember Taverna 1.4 might notice that this is something we've had hidden in the shelves for a bit, we hope that as we bring it back this time it will be a bit more usable as an alternative way to build workflows. You will still be able to flip to the "Graphical" tab for the classic non-interactive (but usually prettier) diagram.
The t2 plugin will now be a bit more usable with support for BioMoby, we've also made the BioMart support use streaming so that as soon as the first row of the BioMart result set has been received, it will immediately be pushed down to the next step in the workflow. Further on, the results of those operations will also be streamed on immediately before the full list is processed. For large datasets this should significantly improve the execution time of the workflow.
The t2 plugin also now has a graphical representation of the workflow as it's running, with progress bars. Note that when doing streaming the progress bar can look a bit weird as results are coming in – say initially processor 2 has finished 4 of 5 items, but 5 new ones comes in from above, then the progress bar will jump from 80% (4/5) to 40% (4/10). But the cool thing is that you will be able to almost see the data as it's flowing through, almost like pipes with pumps between them. We're planning to add more features to this view, say to let you click on a processor as it's running and have a look, perhaps tweak some parameters or go deeper in detail.
After the 1.7.1 release, which we're predicting will be out in about a week (2008-04-07), we'll focus fully on the 2.0 release for June 2008. The 2.0 release will feature the t2 enactor as the core engine, and some graphical updates as well. We'll try to post more on the progress for 2.0 here as we go along.
biomartbiomobyguit2Taverna