<commit_before>﻿(** 
# F# Data: Freebase Provider

The [Freebase graph database](http://www.worldbank.org) contains information on over 
23 million entities, with information on a wide variety of subjects ranging from books 
and movies to historical figures and events to chemical elements, as well as rich 
interconnections between the entities.

The Freebase type provider puts this information at your fingertips, giving you strongly-typed
access to a treasure trove of data. This article provides an introduction. The type provider
is also used on the [Try F#](http://www.tryfsharp.org) web site in the "Data Science" tutorial,
so you can find more examples there. The Visual Studio F# Team Blog also has a series of 4 blog
posts about it [here](http://blogs.msdn.com/b/fsharpteam/archive/2012/09/21/the-f-3-0-freebase-type-provider-sample-integrating-internet-scale-data-sources-into-a-strongly-typed-language.aspx)
and you can watch a recorded demo by <NAME> [here](http://skillsmatter.com/podcast/scala/an-informal-deep-dive-with-don-syme-the-freebase-type-provider)

## Introducing the provider

The following example loads the `FSharp.Data.dll` library (in F# Interactive), 
initializes a connection to Freebase using the `GetDataContext` method and then
retrieves the Chemical Elements, including Hydrogen in three lines of code!
*)

#r "../bin/FSharp.Data.dll"
open FSharp.Data

let data = FreebaseData.GetDataContext()
let elements = data.``Science and Technology``.Chemistry.``Chemical Elements`` |> Seq.toList
let hydrogen = data.``Science and Technology``.Chemistry.``Chemical Elements``.Individuals.Hydrogen

(**
You can now explore the Freebase data schema by using “.” after "data" or "hydrogen".

There is a lot of different data available on Freebase, and you can use it for all kinds of purposes.
Here is an example that generates realistic test data:
*)

let randomElement = 
    let random = new System.Random()
    fun (arr : string[]) -> arr.[random.Next(arr.Length)]

let surnames = 
    data.Society.People.``Family names``
    |> Seq.truncate 100
    |> Seq.map (fun name -> name.Name)
    |> Array.ofSeq

let firstnames = 
    data.Society.Celebrities.Celebrities
    |> Seq.truncate 100
    |> Seq.map (fun celeb -> celeb.Name.Split([|' '|]).[0])
    |> Array.ofSeq

let testNames = seq {
    while true do
        yield (randomElement firstnames) + " " + (randomElement surnames)
}

testNames |> Seq.take 10 |> Seq.toArray

(**
Some of the Freebase type provider features:

* Many queries are translated efficiently into the MQL language. Those that can't execute on the client side by default.
* A selection of sample individuals is given under the "Individuals" entry for each collection of objects. This allows you to program against stroingly named individual such as Hydrogen or ``<NAME>``
* Freebase features such as approximate counts are supported
* The implementation uses the latest Freebase API 
* Image URLs are provided with GetImages(), and the first image is provided using the MainImage property
* Snapshot dates for Freebase are supported
* Optional client-side caching of schema information makes type checking quick and efficient
* API keys are supported
* Units of measure are supported

## Freebase Queries

The Freebase provider includes some support for query translation from F# 3.0 LINQ queries to MQL.
This means you can write queries in F# 3.0 with auto-completion and strong typing, and still execute 
efficiently on the server, at least for the queries translated to MQL .

In addition to the standard F# 3.0 query operators, `FSharp.Data.FreebaseOperators` defines three more
Freebase specific operators: `ApproximatelyMatches`, `ApproximatelyOneOf` and `ApproximateCount`.
Here are some sample queries that are translated fully to MQL:
*)

open FSharp.Data.FreebaseOperators

// Name some sub-domains of data
let biology = data.``Science and Technology``.Biology
let computers = data.``Science and Technology``.Computers
let chemistry = data.``Science and Technology``.Chemistry
let astronomy = data.``Science and Technology``.Astronomy
let books = data.``Arts and Entertainment``.Books
 
/// Get the names of the US presidents
let presidents = 
    query { for e in data.Society.Government.``US Presidents`` do 
            select e.Name } 
    |> Seq.toList
 
/// Count the stars listed in the database
let numberOfStars = astronomy.Stars.Count()
 
/// The name and distances of stars which have a distance recorded.
let someStarDistances = 
    query { for e in astronomy.Stars do 
            where e.Distance.HasValue
            select (e.Name, e.Distance) } 
        |> Seq.toList
 
/// Get the stars in the database sorted by proximity to earth
let starsSortedByProximityToEarth = 
    query { for e in astronomy.Stars do 
            sortBy e.Distance.Value
            take 10
            select e } 
        |> Seq.toList
 
    /// Get some stars close to Earth
let getSomeCloseStars = 
    query { for e in astronomy.Stars do 
            where (e.Distance.Value < 4.011384e+18<_>)
            select e } 
        |> Seq.toList
 
    /// Get the first 10 books matching a user string. 
let topBooksWithNameContaining (s:string) = 
    query { for book in data.``Arts and Entertainment``.Books.Books do
            where (book.Name.ApproximatelyMatches s)
            take 10 
            select book.Name }
 
topBooksWithNameContaining "1984" |> Seq.toList

(**

## Providing an API key

The Freebase API is rate limited, and initially you are using some quota available for debugging purposes.
If you get the (403) Forbidden error, then this shows you are hitting rate limitations. 
Quickly you will need an API key with the Freebase service enabled. This gives you 100,000 requests/day. 
The F# Data Library also provides the `FreebaseDataProvider` type which allows you to specify several parameters,<commit_msg>Describe SendingRequest event in Freebase docs<commit_after>﻿(** 
# F# Data: Freebase Provider

The [Freebase graph database](http://www.worldbank.org) contains information on over 
23 million entities, with information on a wide variety of subjects ranging from books 
and movies to historical figures and events to chemical elements, as well as rich 
interconnections between the entities.

The Freebase type provider puts this information at your fingertips, giving you strongly-typed
access to a treasure trove of data. This article provides an introduction. The type provider
is also used on the [Try F#](http://www.tryfsharp.org) web site in the "Data Science" tutorial,
so you can find more examples there. The Visual Studio F# Team Blog also has a series of 4 blog
posts about it [here](http://blogs.msdn.com/b/fsharpteam/archive/2012/09/21/the-f-3-0-freebase-type-provider-sample-integrating-internet-scale-data-sources-into-a-strongly-typed-language.aspx)
and you can watch a recorded demo by <NAME> [here](http://skillsmatter.com/podcast/scala/an-informal-deep-dive-with-don-syme-the-freebase-type-provider)

## Introducing the provider

The following example loads the `FSharp.Data.dll` library (in F# Interactive), 
initializes a connection to Freebase using the `GetDataContext` method and then
retrieves the Chemical Elements, including Hydrogen in three lines of code!
*)

#r "../bin/FSharp.Data.dll"
open FSharp.Data

let data = FreebaseData.GetDataContext()
let elements = data.``Science and Technology``.Chemistry.``Chemical Elements`` |> Seq.toList
let hydrogen = data.``Science and Technology``.Chemistry.``Chemical Elements``.Individuals.Hydrogen

(**
You can now explore the Freebase data schema by using “.” after "data" or "hydrogen".

There is a lot of different data available on Freebase, and you can use it for all kinds of purposes.
Here is an example that generates realistic test data:
*)

let randomElement = 
    let random = new System.Random()
    fun (arr : string[]) -> arr.[random.Next(arr.Length)]

let surnames = 
    data.Society.People.``Family names``
    |> Seq.truncate 100
    |> Seq.map (fun name -> name.Name)
    |> Array.ofSeq

let firstnames = 
    data.Society.Celebrities.Celebrities
    |> Seq.truncate 100
    |> Seq.map (fun celeb -> celeb.Name.Split([|' '|]).[0])
    |> Array.ofSeq

let testNames = seq {
    while true do
        yield (randomElement firstnames) + " " + (randomElement surnames)
}

testNames |> Seq.take 10 |> Seq.toArray

(**
Some of the Freebase type provider features:

* Many queries are translated efficiently into the MQL language. Those that can't execute on the client side by default.
* A selection of sample individuals is given under the "Individuals" entry for each collection of objects. This allows you to program against stroingly named individual such as Hydrogen or ``<NAME>``
* Freebase features such as approximate counts are supported
* The implementation uses the latest Freebase API 
* Image URLs are provided with GetImages(), and the first image is provided using the MainImage property
* Snapshot dates for Freebase are supported
* Optional client-side caching of schema information makes type checking quick and efficient
* API keys are supported
* Units of measure are supported

## Freebase Queries

The Freebase provider includes some support for query translation from F# 3.0 LINQ queries to MQL.
This means you can write queries in F# 3.0 with auto-completion and strong typing, and still execute 
efficiently on the server, at least for the queries translated to MQL .

In addition to the standard F# 3.0 query operators, `FSharp.Data.FreebaseOperators` defines three more
Freebase specific operators: `ApproximatelyMatches`, `ApproximatelyOneOf` and `ApproximateCount`.
Here are some sample queries that are translated fully to MQL:
*)

open FSharp.Data.FreebaseOperators

// Name some sub-domains of data
let biology = data.``Science and Technology``.Biology
let computers = data.``Science and Technology``.Computers
let chemistry = data.``Science and Technology``.Chemistry
let astronomy = data.``Science and Technology``.Astronomy
let books = data.``Arts and Entertainment``.Books
 
/// Get the names of the US presidents
let presidents = 
    query { for e in data.Society.Government.``US Presidents`` do 
            select e.Name } 
    |> Seq.toList
 
/// Count the stars listed in the database
let numberOfStars = astronomy.Stars.Count()
 
/// The name and distances of stars which have a distance recorded.
let someStarDistances = 
    query { for e in astronomy.Stars do 
            where e.Distance.HasValue
            select (e.Name, e.Distance) } 
        |> Seq.toList
 
/// Get the stars in the database sorted by proximity to earth
let starsSortedByProximityToEarth = 
    query { for e in astronomy.Stars do 
            sortBy e.Distance.Value
            take 10
            select e } 
        |> Seq.toList
 
    /// Get some stars close to Earth
let getSomeCloseStars = 
    query { for e in astronomy.Stars do 
            where (e.Distance.Value < 4.011384e+18<_>)
            select e } 
        |> Seq.toList
 
    /// Get the first 10 books matching a user string. 
let topBooksWithNameContaining (s:string) = 
    query { for book in data.``Arts and Entertainment``.Books.Books do
            where (book.Name.ApproximatelyMatches s)
            take 10 
            select book.Name }
 
topBooksWithNameContaining "1984" |> Seq.toList

(**

You can see the MQL queries that are being made by subscribing to the `SendingRequest` event
*)

data.DataContext.SendingRequest.Add (fun e -> printfn "request: %A" e.RequestUri)

(**

## Providing an API key

The Freebase API is rate limited, and initially you are using some quota available for debugging purposes.
If you get the (403) Forbidden error, then this shows you are hitting rate limitations. 
Quickly you will need an API key with the Freebase service enabled. This gives you 100,000 requests/day. 
The F# Data Library also provides the `FreebaseDataProvider` type which allows you to specify several parameters,
{'fraction_non_alphanumeric': 0.0816105368456152, 'fraction_numerical': 0.009336445481827277, 'mean_word_length': 3.6826697892271665, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5407983', 'n_tokens_mistral': 3554, 'n_tokens_neox': 3237, 'n_words': 1405}