<commit_before># freebase-likq: hosting LIKQ queries on the freebase data set

In this demo we show how to host LIKQ queries on the freebase data set. 
In general, hosting LIKQ queries over a data set involves the following steps:

- Design the storage schema, and import the raw data into Graph Engine storage.
- Tag data fields in the schema file (TSL) with [GraphEdge] attribute so that LIKQ recognizes these TSL fields as graph edges.
- Create an application referencing GraphEngine.LIKQ and GraphEngine.Core packages (GraphEngine.Core will be automatically installed as a dependency).
- In the application, register the LIKQ module into a TrinityServer before starting it.
- For a particular data set, one may want to do some customization (polymorphic type system, custom indexing backend etc.). See `Program.cs` for details.

Please refer to `GraphEngine.DataImporter` sample project in the `experimental` branch to see how to automatically generate schema file and import the data.
For brevity, in this demo we only focus on how to host LIKQ queries with prebuilt data image/TSL script (The data/TSL script are imported with the importer).

A particularlly interesting feature of this demo is that it supports polymorphism for an entity (e.g. a `people_person` could also be a `film_actor`).
This is achieved by alternating the way that LIKQ accesses the storage. By default, when LIKQ visits a graph node, it directly allocates a TSL accessor specifically
built for a type (e.g. `people_person_Accessor`). Instead, we store each facet of an entity into one cell, and create a root cell that holds metadata of the entity.
We then create an adapter that implements the generic `ICellAccessor` interface, but not bound to a particular type.
When LIKQ requests a field from this adapter, it first queries the root cell to validate the operation, then re-routes the data access operation to one of the "physical" accessors.

A sample index service that interfaces with the LIKQ module is provided.
When query constraints are specified in a query, the LIKQ module calls a registered index service to retrieve vertices satisfying the constraints.
The constraints are specified in the match object, which is a json object. LIKQ itself does not specify the DSL syntax for the match object so here
we eatablish a simple one, which routes the constraints into SQLite.

## Building and running the demo

In the `master`/`dev` branch, simply build the freebase-likq solution. In the `experimental` branch,
it requires the experimental GraphEngine.Core and GraphEngine.LIKQ packages, so you should build them first and put them
into a local nuget package source, so that Nuget package manager can restore these packages during a build.

After the solution is built successfully, run `freebase-likq.exe`, it will automatically download the Freebase data image, build the SQLite index and start serving.
To enable the HTTP endpoints, either elevate the program, or grant the current user account the permission to listen to port 80: `netsh http add urlacl url=http://+:80/ user=Domain\username`.<commit_msg>updates freebase-likq readme<commit_after># freebase-likq: hosting LIKQ queries on the freebase data set

In this demo, we demonstrate how to serve LIKQ queries for the
[Freebase](https://en.wikipedia.org/wiki/Freebase) data set.  It
generally takes the following 3 steps to serve LIKQ queries for a data
set:

- Specify the storage schema for the data in [TSL](https://www.graphengine.io/docs/manual/TSL/index.html). Remember to tag graph edges using [GraphEdge] attribute in the TSL script.
- Import data into Graph Engine.
- Create a C# project referencing GraphEngine.LIKQ and register the LIKQ module into a TrinityServer (refer to the sample code for an example).

For brevity, in this demo we focus on how to serve LIKQ queries with a
prebuilt data image.  Please refer to the `GraphEngine.DataImporter`
sample project in the `experimental` branch to see how to
automatically generate a schema file and import data.

A sample index service based on SQLite that interacts with the LIKQ
module is provided.  When query constraints are specified in a LIKQ
query, the LIKQ module calls a registered index service to retrieve
the graph nodes satisfying the constraints.  The constraints are
specified in the _match_ object, which is a json object. LIKQ itself
does not parse the _match_ object. The constraints specified by the
_match_ object will be relayed to the index backend and parsed by the
index backend.

## Building and running the demo

After the freebase-likq solution is built successfully, run
`freebase-likq.exe`. It will automatically download the Freebase data
image, build the SQLite index and start serving LIKQ.  To enable the
HTTP endpoints, either run the program as administrator, or grant the
current user the permission to listen to port 80: `netsh http add
urlacl url=http://+:80/ user=Domain\username`.

Now you can query Freebase via LIKQ. Here is a quick example:


It is worth noting that Freebase entities are usually of multiple
types. For example, a person might be a book author and a film actor
at the same time.

{'fraction_non_alphanumeric': 0.043689320388349516, 'fraction_numerical': 0.0017475728155339806, 'mean_word_length': 4.605005440696409, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '644985', 'n_tokens_mistral': 1307, 'n_tokens_neox': 1271, 'n_words': 786}