<commit_before>---
layout: post
title: YQL mashup with GWT
excerpt: In this post I build a simple mashup with Google Web Toolkit that uses Yahoo Query language to retrieve pictures from flickr and render them in a small GWT component.
---
p. Yahoo came up with YQL a few weeks ago, and I think it is a very interesting idea. It is basically a SQL-like language that allows developers to retrieve structured data from multiple Yahoo web services without having to deal with each service particularities; the developer can query any service in a generic way, just like he would do retrieving rows and columns from a SQL database.

An example of YQL query to retrieve the 5 first results returned by the search engine with the keyword "pizza" would be:

{% highlight sql %}
select title, abstract from search.web where query="pizza"
{% endhighlight %}

which returns a response in the format that you specify (either XML or JSON) that includes some metadata about the query itself and the actual requested results in a format like this (JSON):

{% highlight javascript %}
"results": {
    "result": [
    {
     "abstract": "Official site for the international <strong>pizza</strong> chain, with online ordering, etc.",
     "title": "<strong>Pizza</strong> Hut"
    },
    {
     "abstract": "<strong>...</strong> site for the delivery chain offering thin crust, deep dish, etc.",
     "title": "Domino's <strong>Pizza</strong>"
    },
    ...
    ]
}
{% endhighlight %}

As you can see it is a very intuitive way to interact with web services (if only they were not all from Yahoo) for everybody that knows a bit of SQL.

If you want to try other queries or see examples of its use with other services you can use the "YQL Console":http://developer.yahoo.com/yql/console/.

h2. Querying YQL with Google Web Toolkit

!http://www.sergimansilla.com/blog/wp-content/uploads/2008/11/screenshot-2-182x300.png!
So how would we query YQL with GWT? As it turns out, very easily. I am going to create a GWT component that queries flickr through YQL and retrieves the first 9 occurrences related to the keyword entered in a textbox and presents them in a Grid, like the one on the left.

h3. Authentication

YQL uses 2-legged OAuth authentication like most OpenSocial containers, so we will have to have this into account when making the request. Fortunately there are good OAuth libraries available for every language. In my example I will use the ones made by Google, available at "http://oauth.googlecode.com":http://oauth.googlecode.com. I copied them to the /public folder and I referenced them from the module xml file instead of loading them directly in the html, which is more GWT-ish.

When I browsed the web for OAuth libraries I stumbled upon "<NAME>'s blog":http://paul.donnelly.org, who made a simple Javascrifunction for processing the parameters returned by OAUth libraries in the fashion that YQL wants, so instead of doing it with pure Java I took advantage of the toolkit's JSNI classes and I just made a dumb OAuth class with a single static container method makeSignedRequest that wraPaul's very slightly modified function so that it references the OAuth object properly.

h3. Calling back

Note: This section is explained in much more detail at "<NAME>'s JSON tutorial":http://code.google.com/support/bin/answer.py?answer=65632&amp;topic=11368.

Now that we have an OAuth class that will deal with the authentication, we only have to worry about the actual communication. Before getting our hands dirty with it, I will explain a bit some particularities of YQL and GWT that are necessary to get the mashup working properly.

GWT doesn't directly allow fetching JSON data from an external domain for security reasons (any JSON data is a Javascriobject after all, and although it should be used only for data transport, it can contain executable code as well), so I will use a "bridge method":http://code.google.com/support/bin/answer.py?answer=55954&amp;topic=10213 setup to fetch it from Yahoo's domain.

When we send a request to YQL through OAuth, we send some important parameters with it: our personal YQL key and YQL secret (generated when we request a new application account), and the URL in which we specify the encoded query, the response format we want and the callback function name that YQL will use in its response. The request method looks like this:

{% highlight java %}
public void executeQuery(String keyword)
{
    String callback = this.reserveCallback();

    // Generate a function in the page with the known name we just created
    flyqlr.setup(this, callback);

    // Encode the URL using GWT's |URL.encode| and replace remaining unencoded characters
    String query = URL.encode("select * from flickr.photos.search where text = '" + keyword + "' limit 9")
                  .replaceAll("'", "%22").replaceAll("=", "%3D");

    // Makes the request using OAuth, which calls our just created callback
    // and adds the response as a scrito the page
    this.addScript(OAuth.makeSignedRequest(
        flyqlr.Y_KEY, flyqlr.Y_SECRET, flyqlr.YQL_URL + "q=" + query + "&amp;format=json" + "&amp;callback=" + callback));
}

{% endhighlight %}

The first thing that the method does is to generate a unique name for our future function and pass it as a parameter to the setup static method, which is the JSNI "bridge" method mentioned above, and that looks like this:

{% highlight java %}
public native static void setup(flyqlr f, String callback) /*-{
  $wnd[callback] = function(data) {
    f.@com.sergi.client.flyqlr::process(Lcom/google/gwt/core/client/JavaScriptObject;)(data);
  }
}-*/;
{% endhighlight %}

This method creates a new function inside the window object ($wnd in JSNI slang) named after the callback parameter. This function calls the Java method process with a parameter data which contains the JSON string returned by YQL. When we know that our callback exists in the window we can safely compose the query and make the OAuth request. Notice that the latest parameter of the request string is callback, which tells YQL services our callback function's name.

The addScrimethod just creates a scritag and injects its argument as its content and appends the tag to the page which executes its contents straight away, thus executing our callback.

h3. Processing the response
<commit_msg>Small layout change in gwt post<commit_after>---
layout: post
title: YQL mashup with GWT
excerpt: In this post I build a simple mashup with Google Web Toolkit that uses Yahoo Query language to retrieve pictures from flickr and render them in a small GWT component.
---
p. Yahoo came up with YQL a few weeks ago, and I think it is a very interesting idea. It is basically a SQL-like language that allows developers to retrieve structured data from multiple Yahoo web services without having to deal with each service particularities; the developer can query any service in a generic way, just like he would do retrieving rows and columns from a SQL database.

An example of YQL query to retrieve the 5 first results returned by the search engine with the keyword "pizza" would be:

{% highlight sql %}
select title, abstract from search.web where query="pizza"
{% endhighlight %}

which returns a response in the format that you specify (either XML or JSON) that includes some metadata about the query itself and the actual requested results in a format like this (JSON):

{% highlight javascript %}
"results": {
    "result": [
    {
     "abstract": "Official site for the international <strong>pizza</strong> chain, with online ordering, etc.",
     "title": "<strong>Pizza</strong> Hut"
    },
    {
     "abstract": "<strong>...</strong> site for the delivery chain offering thin crust, deep dish, etc.",
     "title": "Domino's <strong>Pizza</strong>"
    },
    ...
    ]
}
{% endhighlight %}

As you can see it is a very intuitive way to interact with web services (if only they were not all from Yahoo) for everybody that knows a bit of SQL.

If you want to try other queries or see examples of its use with other services you can use the "YQL Console":http://developer.yahoo.com/yql/console/.

h2. Querying YQL with Google Web Toolkit

!http://www.sergimansilla.com/blog/wp-content/uploads/2008/11/screenshot-2-182x300.png!
So how would we query YQL with GWT? As it turns out, very easily. I am going to create a GWT component that queries flickr through YQL and retrieves the first 9 occurrences related to the keyword entered in a textbox and presents them in a Grid, like the one on the left.

h3. Authentication

YQL uses 2-legged OAuth authentication like most OpenSocial containers, so we will have to have this into account when making the request. Fortunately there are good OAuth libraries available for every language. In my example I will use the ones made by Google, available at "http://oauth.googlecode.com":http://oauth.googlecode.com. I copied them to the /public folder and I referenced them from the module xml file instead of loading them directly in the html, which is more GWT-ish.

When I browsed the web for OAuth libraries I stumbled upon "<NAME>nelly's blog":http://paul.donnelly.org, who made a simple Javascrifunction for processing the parameters returned by OAUth libraries in the fashion that YQL wants, so instead of doing it with pure Java I took advantage of the toolkit's JSNI classes and I just made a dumb OAuth class with a single static container method makeSignedRequest that wraPaul's very slightly modified function so that it references the OAuth object properly.

h3. Calling back

Note: This section is explained in much more detail at "Dan Morrill's JSON tutorial":http://code.google.com/support/bin/answer.py?answer=65632&amp;topic=11368.

Now that we have an OAuth class that will deal with the authentication, we only have to worry about the actual communication. Before getting our hands dirty with it, I will explain a bit some particularities of YQL and GWT that are necessary to get the mashup working properly.

GWT doesn't directly allow fetching JSON data from an external domain for security reasons (any JSON data is a Javascriobject after all, and although it should be used only for data transport, it can contain executable code as well), so I will use a "bridge method":http://code.google.com/support/bin/answer.py?answer=55954&amp;topic=10213 setup to fetch it from Yahoo's domain.

When we send a request to YQL through OAuth, we send some important parameters with it: our personal YQL key and YQL secret (generated when we request a new application account), and the URL in which we specify the encoded query, the response format we want and the callback function name that YQL will use in its response. The request method looks like this:

{% highlight java %}
public void executeQuery(String keyword)
{
    String callback = this.reserveCallback();

    // Generate a function in the page with the known name we just created
    flyqlr.setup(this, callback);

    // Encode the URL using GWT's |URL.encode| and replace remaining unencoded characters
    String query = URL.encode("select * from flickr.photos.search where text = '" + keyword + "' limit 9")
                  .replaceAll("'", "%22").replaceAll("=", "%3D");

    // Makes the request using OAuth, which calls our just created callback
    // and adds the response as a scrito the page
    this.addScript(OAuth.makeSignedRequest(
        flyqlr.Y_KEY, flyqlr.Y_SECRET, flyqlr.YQL_URL 
            + "q=" + query + "&amp;format=json" + "&amp;callback=" + callback));
}

{% endhighlight %}

The first thing that the method does is to generate a unique name for our future function and pass it as a parameter to the setup static method, which is the JSNI "bridge" method mentioned above, and that looks like this:

{% highlight java %}
public native static void setup(flyqlr f, String callback) /*-{
  $wnd[callback] = function(data) {
    f.@com.sergi.client.flyqlr::process(Lcom/google/gwt/core/client/JavaScriptObject;)(data);
  }
}-*/;
{% endhighlight %}

This method creates a new function inside the window object ($wnd in JSNI slang) named after the callback parameter. This function calls the Java method process with a parameter data which contains the JSON string returned by YQL. When we know that our callback exists in the window we can safely compose the query and make the OAuth request. Notice that the latest parameter of the request string is callback, which tells YQL services our callback function's name.

The addScrimethod just creates a scritag and injects its argument as its content and appends the tag to the page which executes its contents straight away, thus executing our callback.

h3. Processing the response

{'fraction_non_alphanumeric': 0.06637829335430594, 'fraction_numerical': 0.006920959496657491, 'mean_word_length': 4.148178137651822, 'pattern_counts': {'":': 22, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6977544', 'n_tokens_mistral': 3390, 'n_tokens_neox': 3201, 'n_words': 1810}