<commit_before>* Grafter

Exploring the use of Clojure for RDFizing, DSL's, RDF tooling and more
tightly integrating with triple stores.

Grafter has a number of stated aims:

1) To show that Clojure could provide us with a number of platform
   benefits:

   - Easy access to Java tooling & API's, including the ability to
     talk direct to underlying triple store implementations.
   - Simplified infrastructure due to excellent concurrency support
     etc.
   - Potentially better performance by
     + Utilising local disk backed, indexed triple stores with no
       additional infrastructure overhead
     + Using a more performant language
     + Potentially skipping unnecessary serialization/parsing steps
   - Being able to create and manage logical graphs/endpoints more
     easily.

2) To show that Clojure is a near perfect tool for building an
   RDFization DSL which has a number of desirable properties that
   would be harder to achieve with more conventional approaches.

* RDFizing

In terms of RDFization the core idea is:

1) To build up a decent library of low level functions for common
   RDFing tasks.

2) To build a DSL with a simplified, consistent syntax that can be
   easily targetted by tooling etc...  The idea here is that the DSL's
   syntax will be built purely in terms of [[https://github.com/edn-format/edn][Extensible Data Notation]].

   (EDN is sort of like JSON, except it's syntactically Clojure,
   supports more data types and collections, and is extensible.
   API's for parsing and writing EDN are available in lots of
   languages including [[https://github.com/relevance/edn-ruby][Ruby]] and [[https://github.com/shaunxcode/jsedn][Javascript]])

   Above the syntactic layer Grafter should try and provide a language
   of operations that are simple enough for basic end user tooling,
   whilst ensuring sane (non-turing complete) semantics.

   As the grafter DSL snippets will also be valid Clojure programs,
   DSL forms will be literally copy/pastable inside arbitrary Clojure
   programs.

3) Grafter should aim to be a componentized workflow language, where
   logging, error reporting and more advanced features such as
   debugging, tracing and the configuration of concrete
   implementations can be easily applied by either the environment or
   a user to arbitrary pipelines without having to modify the pipeline
   itself.

We refer to the lower-level of grafter as the Grafter API and the
higher level as the grafter DSL.  Sometimes functions may be both part
of the DSL and the API so the distinction is not necessarily always
clear cut.

** Worked Example

Grafter can be thought of a number of separate DSLs that are each
targeted at specialised tasks that can then be seamlessly tied
together.

The main DSL's are:

- The Cell Conversion DSL :: A mini DSL of composable functions, and
     higher-order functions which take a single argument and validate
     and convert its type.
- The Table Operation DSL :: This DSL tries to map cleanly to
     intuitive spreadsheet like operations.  The idea is that the bulk
     of the transformation and wiring required can be done here.
- Triple Templates DSL :: This DSL receives each processed row as an
     argument, and returns a sequence of Statements.
- Filtering & Validation DSL :: This is pretty much absent apart from
     a filtering clause at this stage.

*** User Definable Functions

It is envisaged that users will need to define their own functions.

Firstly a UI could be built that would allow simple mapping functions
to be defined.  For example URIifying facility types can be expressed
like this with the urban prefixer:

#+BEGIN_SRC clojure
(def uriify-facility {"Museums" (urban "Museum")
                      "Arts" (urban "ArtsCentre")
                      "Community Facility" (urban "CommunityFacility")
                      "Libraries" (urban "Library")
                      "Music" (urban "MusicVenue")
                      "Sport Centres" (urban "SportsCentre")})
#+END_SRC

At the next level of complexity, users can supply their own javascript
functions which can be used within the pipeline.  These functions are
compiled and wrapped with a small Clojure function so they can be used
seamlessly in the pipeline.

#+BEGIN_SRC clojure
(def slugify-facility
  (js-fn "function(name) {
              var lower = name.toLowerCase();
              return lower.replace(/\\ /g, '-');
         }"))
#+END_SRC

Finally we can obviously also define functions in Clojure:

#+BEGIN_SRC clojure
(defn date-slug [date]
  (str (.getYear date) "-" (.getMonthOfYear date) "/"))
#+END_SRC

*** Extract : Cell Conversion DSL

This DSL is currently monadic in order to allow uniform (swapable
logic) to occur between the functions being composed.  The main
utility of this right now is that it means you can more easily import
standard clojure/java functions into the API and adapt them
independently to the cell parsing environment.  For example parse-int
does not need to implement error handling for empty strings etc, as
that can be provided by the monad (here blank-m).

I am still a little unsure about this part of the API, but it works
quite well.

Things to note about it:

- Each function takes one argument
- Each line binds a name to a function.
- Functions are composed pointfree (the argument passing is invisible)
- Sometimes we call a "function factory" to build us a function, to do
  the job we want, e.g. replacing "," with the empty string, or
  mapping the empty string to "0".
- You can adjust what happens between functions such as the error
  handling strategy by wrapping it with a different monad.  The
  closest binding scope wins.

Future ideas:

- We could make the DSL more terse and expressive by effectively also
  using these function names to identify the columns in the DSL.
- You can interpret these functions in several ways depending on
  context, i.e. you could run them across the cells to display
  errors/warnings or by swapping the monad you could use the same
  functions to do the type conversions as we do here.

#+BEGIN_SRC clojure
  (with-monad blank-m
    (let [rdfstr                    (lift-1 (fn [str] (s str :en)))
          replace-comma             (lift-1 (replacer "," ""))
          trim                      (lift-1 clojure.string/trim)
          parse-attendance          (with-monad identity-m (m-chain [(lift-1 (mapper {"" "0"}))
                                                                     replace-comma
                                                                     trim
                                                                     parse-int]))
          parse-year                (m-chain [trim replace-comma parse-int])
          convert-month             (m-chain [trim
                                              (lift-1 clojure.string/lower-case)
                                              (lift-1 {"january" 1 "jan" 1 "1" 1
                                                       "february" 2 "feb" 2 "2" 2
                                                       "march" 3 "mar" 3 "3" 3
                                                       "april" 4 "apr" 4 "4" 4
                                                       "may" 5 "5" 5
                                                       "june" 6 "jun" 6 "6"  6
                                                       "july" 7 "jul" 7 "7"  7
                                                       "august" 8 "aug" 8 "8" 8
                                                       "september" 9 "sep" 9 "sept" 9 "9"  9
                                                       "october" 10 "oct" 10 "10" 10
                                                       "november" 11 "nov" 11 "11" 11
                                                       "december" 12 "dec" 12 "12" 12
                                                       })])
          convert-year              (m-chain [trim parse-int date-time])
          address-line              (m-chain [trim rdfstr])
          city                      (m-chain [trim rdfstr])
          post-code                 (m-chain [trim rdfstr])
          uriify-pcode              (m-chain [trim
                                              (lift-1 (replacer " " ""))
                                              (lift-1 clojure.string/upper-case)
                                              (lift-1 (prefixer "http://data.ordnancesurvey.co.uk/id/postcodeunit/"))])
          url                       (lift-1 #(java.net.URL. %))

          prefix-monthly-attendance (m-chain [(lift-1 date-slug)
                                              (lift-1 (prefixer "/community-facility/"))])
          prefix-facility           (prefixer "http://linked.glasgow.gov.uk/data/facility_attendance")]

  ;; table conversion code here.
  ))
#+END_SRC

*** Transform : Table conversion DSL

The code below is part of my original CSV table DSL, which has had a
few new functions added.  It has been designed to work cleanly with
Clojure's thread-first macro =->= though we might want to extend this
at some point to make it more monadic.

Note this DSL is not tied to the concrete implementation of CSV files,
but instead can operate on any arbitrary sequence of vectors.  Meaning
we only need to write =parse-shape-file= to generate a (lazy) sequence
of vectors and it will also work.

#+BEGIN_SRC clojure
     ;; ^--- let bindings
            (-> (parse-csv "./test-data/glasgow-life-facilities.csv")
                (drop-rows 1)
                (swap {3 4})
                (mapc [uriify-facility _ parse-attendance parse-year convert-month address-line city post-code url])
                (derive-column uriify-pcode 7)
                (fuse date-time 3 4)
                (derive-column prefix-monthly-attendance 3)
                (derive-column slugify-facility 1)
                (fuse str 9 10)
                (derive-column prefix-facility 9))
#+END_SRC

You can read this DSL as applying the specified operations in order
(top to bottom) to a whole CSV file.  At each stage in the pipeline it
is as if a whole new CSV file is there, however underneath it is built
entirely out of lazy sequences; which means all of this is achieved in
only one iteration of the whole file; rather than the 10 iterations
you might typically expect.  To prove it consuming the whole sequence
takes 45ms:

#+BEGIN_SRC clojure
grafter.rdf-examples> (time (dorun (make-life-facilities)))
"Elapsed time: 45.38 msecs"
#+END_SRC

But if we take just one item out of it, it only has to process 1 row
so it takes 2ms:

#+BEGIN_SRC clojure
grafter.rdf-examples> (time (dorun (take 1 (make-life-facilities))))
"Elapsed time: 2.059 msecs"
#+END_SRC

Note the interesting thing here is that we can specify how much data
we want to consume outside of the core algorithm!  This allows us to
trivially use the same code to preview the first 50 rows.  The core
algorithm itself never implies how much work it will actually do.

Ok... So what is the code actually doing?

#+BEGIN_SRC clojure
     ;; ^--- let bindings
            (-> (parse-csv "./test-data/glasgow-life-facilities.csv")
                (drop-rows 1)
                (swap {3 4})
                (mapc [uriify-facility _ parse-attendance parse-year convert-month address-line city post-code url])
                (derive-column uriify-pcode 7)
                (fuse date-time 3 4)
                (derive-column prefix-monthly-attendance 3)
                (derive-column slugify-facility 1)
                (fuse str 9 10)
                (derive-column prefix-facility 9))
#+END_SRC

First we do the boring stuff we load the file, and skip past
the first row because it's a header row.

Finally we start doing something interesting.  We swap the position of
columns 3 and 4.  Why?  Because our date function date-time expects to
receive the year first, followed by the month, so we can give our
users the necessary power to swap arguments by letting them do so in
the spreadsheet (DSL), rather than in code.

Next up we apply mapc to each row, where mapc takes each of the
functions we defined at the top in our function composition DSL and
applies them to specific columns in the spreadsheet.

These functions perform some initial input validation, and convert the
types from strings into more meaningful values [fn:1].

Next we use =derive-column= to apply a function to an existing column
and put the result in a new column at the end of the spreadsheet.
Here we take the postcode and convert it into a URI.

Currently the DSL has explicitly avoided supporting multiple parameter
function calls.  However it is clear that they are needed, so we need
a constrained way to allow the operation to occur.

=fuse= allows just this.  It takes an arbitrary number of column ids,
here the year and month, and applies each column to the supplied
function as an argument.  You could imagine a simple user interface
would easily allow users to select a function and the columns you want
to apply.  This DSL is ideally the only place we would allow the user
to use multi-argument functions.

Next we use derive-colum and fuse to build two more URI's, one for the
facility and the other for the monthly attendances.  Note that we show
the use of a user supplied javascript functions that we defined
earlier.

*** Transform Templates : Triple Templates Revisited



#+BEGIN_SRC clojure
((graphify [facility-uri name attendance date street-address city postcode website postcode-uri
                    _ observation-uri]
<commit_msg>More docs changes.<commit_after>* Grafter

Exploring the use of Clojure for RDFizing, DSL's, RDF tooling and more
tightly integrating with triple stores.

Grafter has a number of stated aims:

1) To show that Clojure could provide us with a number of platform
   benefits:

   - Easy access to Java tooling & API's, including the ability to
     talk direct to underlying triple store implementations.
   - Simplified infrastructure due to excellent concurrency support
     etc.
   - Potentially better performance by
     + Utilising local disk backed, indexed triple stores with no
       additional infrastructure overhead
     + Using a more performant language
     + Potentially skipping unnecessary serialization/parsing steps
   - Being able to create and manage logical graphs/endpoints more
     easily.

2) To show that Clojure is a near perfect tool for building an
   RDFization DSL which has a number of desirable properties that
   would be harder to achieve with more conventional approaches.

* RDFizing

In terms of RDFization the core idea is:

1) To build up a decent library of low level functions for common
   RDFing tasks.

2) To build a DSL with a simplified, consistent syntax that can be
   easily targetted by tooling etc...  The idea here is that the DSL's
   syntax will be built purely in terms of [[https://github.com/edn-format/edn][Extensible Data Notation]].

   (EDN is sort of like JSON, except it's syntactically Clojure,
   supports more data types and collections, and is extensible.
   API's for parsing and writing EDN are available in lots of
   languages including [[https://github.com/relevance/edn-ruby][Ruby]] and [[https://github.com/shaunxcode/jsedn][Javascript]])

   Above the syntactic layer Grafter should try and provide a language
   of operations that are simple enough for basic end user tooling,
   whilst ensuring sane (non-turing complete) semantics.

   As the grafter DSL snippets will also be valid Clojure programs,
   DSL forms will be literally copy/pastable inside arbitrary Clojure
   programs.

3) Grafter should aim to be a componentized workflow language, where
   logging, error reporting and more advanced features such as
   debugging, tracing and the configuration of concrete
   implementations can be easily applied by either the environment or
   a user to arbitrary pipelines without having to modify the pipeline
   itself.

We refer to the lower-level of grafter as the Grafter API and the
higher level as the grafter DSL.  Sometimes functions may be both part
of the DSL and the API so the distinction is not necessarily always
clear cut.

** Worked Example

Grafter can be thought of a number of separate DSLs that are each
targeted at specialised tasks that can then be seamlessly tied
together.

The main DSL's are:

- The Cell Conversion DSL :: A mini DSL of composable functions, and
     higher-order functions which take a single argument and validate
     and convert its type.
- The Table Operation DSL :: This DSL tries to map cleanly to
     intuitive spreadsheet like operations.  The idea is that the bulk
     of the transformation and wiring required can be done here.
- Triple Templates DSL :: This DSL receives each processed row as an
     argument, and returns a sequence of Statements.
- Filtering & Validation DSL :: This is pretty much absent apart from
     a filtering clause at this stage.

*** User Definable Functions

It is envisaged that users will need to define their own functions.

Firstly a UI could be built that would allow simple mapping functions
to be defined.  For example URIifying facility types can be expressed
like this with the urban prefixer:

#+BEGIN_SRC clojure
(def uriify-facility {"Museums" (urban "Museum")
                      "Arts" (urban "ArtsCentre")
                      "Community Facility" (urban "CommunityFacility")
                      "Libraries" (urban "Library")
                      "Music" (urban "MusicVenue")
                      "Sport Centres" (urban "SportsCentre")})
#+END_SRC

At the next level of complexity, users can supply their own javascript
functions which can be used within the pipeline.  These functions are
compiled and wrapped with a small Clojure function so they can be used
seamlessly in the pipeline.

#+BEGIN_SRC clojure
(def slugify-facility
  (js-fn "function(name) {
              var lower = name.toLowerCase();
              return lower.replace(/\\ /g, '-');
         }"))
#+END_SRC

Finally we can obviously also define functions in Clojure:

#+BEGIN_SRC clojure
(defn date-slug [date]
  (str (.getYear date) "-" (.getMonthOfYear date) "/"))
#+END_SRC

*** Extract : Cell Conversion DSL

This DSL is currently monadic in order to allow uniform (swapable
logic) to occur between the functions being composed.  The main
utility of this right now is that it means you can more easily import
standard clojure/java functions into the API and adapt them
independently to the cell parsing environment.  For example parse-int
does not need to implement error handling for empty strings etc, as
that can be provided by the monad (here blank-m).

I am still a little unsure about this part of the API, but it works
quite well.

Things to note about it:

- Each function takes one argument
- Each line binds a name to a function.
- Functions are composed pointfree (the argument passing is invisible)
- Sometimes we call a "function factory" to build us a function, to do
  the job we want, e.g. replacing "," with the empty string, or
  mapping the empty string to "0".
- You can adjust what happens between functions such as the error
  handling strategy by wrapping it with a different monad.  The
  closest binding scope wins.

Future ideas:

- We could make the DSL more terse and expressive by effectively also
  using these function names to identify the columns in the DSL.
- You can interpret these functions in several ways depending on
  context, i.e. you could run them across the cells to display
  errors/warnings or by swapping the monad you could use the same
  functions to do the type conversions as we do here.

#+BEGIN_SRC clojure
  (with-monad blank-m
    (let [rdfstr                    (lift-1 (fn [str] (s str :en)))
          replace-comma             (lift-1 (replacer "," ""))
          trim                      (lift-1 clojure.string/trim)
          parse-attendance          (with-monad identity-m (m-chain [(lift-1 (mapper {"" "0"}))
                                                                     replace-comma
                                                                     trim
                                                                     parse-int]))
          parse-year                (m-chain [trim replace-comma parse-int])
          convert-month             (m-chain [trim
                                              (lift-1 clojure.string/lower-case)
                                              (lift-1 {"january" 1 "jan" 1 "1" 1
                                                       "february" 2 "feb" 2 "2" 2
                                                       "march" 3 "mar" 3 "3" 3
                                                       "april" 4 "apr" 4 "4" 4
                                                       "may" 5 "5" 5
                                                       "june" 6 "jun" 6 "6"  6
                                                       "july" 7 "jul" 7 "7"  7
                                                       "august" 8 "aug" 8 "8" 8
                                                       "september" 9 "sep" 9 "sept" 9 "9"  9
                                                       "october" 10 "oct" 10 "10" 10
                                                       "november" 11 "nov" 11 "11" 11
                                                       "december" 12 "dec" 12 "12" 12
                                                       })])
          convert-year              (m-chain [trim parse-int date-time])
          address-line              (m-chain [trim rdfstr])
          city                      (m-chain [trim rdfstr])
          post-code                 (m-chain [trim rdfstr])
          uriify-pcode              (m-chain [trim
                                              (lift-1 (replacer " " ""))
                                              (lift-1 clojure.string/upper-case)
                                              (lift-1 (prefixer "http://data.ordnancesurvey.co.uk/id/postcodeunit/"))])
          url                       (lift-1 #(java.net.URL. %))

          prefix-monthly-attendance (m-chain [(lift-1 date-slug)
                                              (lift-1 (prefixer "/community-facility/"))])
          prefix-facility           (prefixer "http://linked.glasgow.gov.uk/data/facility_attendance")]

  ;; table conversion code here.
  ))
#+END_SRC

*** Transform : Table conversion DSL

The code below is part of my original CSV table DSL, which has had a
few new functions added.  It has been designed to work cleanly with
Clojure's thread-first macro =->= though we might want to extend this
at some point to make it more monadic.

Note this DSL is not tied to the concrete implementation of CSV files,
but instead can operate on any arbitrary sequence of vectors.  Meaning
we only need to write =parse-shape-file= to generate a (lazy) sequence
of vectors and it will also work.

#+BEGIN_SRC clojure
     ;; ^--- let bindings
            (-> (parse-csv "./test-data/glasgow-life-facilities.csv")
                (drop-rows 1)
                (swap {3 4})
                (mapc [uriify-facility _ parse-attendance parse-year convert-month address-line city post-code url])
                (derive-column uriify-pcode 7)
                (fuse date-time 3 4)
                (derive-column prefix-monthly-attendance 3)
                (derive-column slugify-facility 1)
                (fuse str 9 10)
                (derive-column prefix-facility 9))
#+END_SRC

You can read this DSL as applying the specified operations in order
(top to bottom) to a whole CSV file.  At each stage in the pipeline it
is as if a whole new CSV file is there, however underneath it is built
entirely out of lazy sequences; which means all of this is achieved in
only one iteration of the whole file; rather than the 10 iterations
you might typically expect.  To prove it consuming the whole sequence
takes 45ms:

#+BEGIN_SRC clojure
grafter.rdf-examples> (time (dorun (make-life-facilities)))
"Elapsed time: 45.38 msecs"
#+END_SRC

But if we take just one item out of it, it only has to process 1 row
so it takes 2ms:

#+BEGIN_SRC clojure
grafter.rdf-examples> (time (dorun (take 1 (make-life-facilities))))
"Elapsed time: 2.059 msecs"
#+END_SRC

Note the interesting thing here is that we can specify how much data
we want to consume outside of the core algorithm!  This allows us to
trivially use the same code to preview the first 50 rows.  The core
algorithm itself never implies how much work it will actually do.

Ok... So what is the code actually doing?

#+BEGIN_SRC clojure
     ;; ^--- let bindings
            (-> (parse-csv "./test-data/glasgow-life-facilities.csv")
                (drop-rows 1)
                (swap {3 4})
                (mapc [uriify-facility _ parse-attendance parse-year convert-month address-line city post-code url])
                (derive-column uriify-pcode 7)
                (fuse date-time 3 4)
                (derive-column prefix-monthly-attendance 3)
                (derive-column slugify-facility 1)
                (fuse str 9 10)
                (derive-column prefix-facility 9))
#+END_SRC

First we do the boring stuff we load the file, and skip past
the first row because it's a header row.

Finally we start doing something interesting.  We swap the position of
columns 3 and 4.  Why?  Because our date function date-time expects to
receive the year first, followed by the month, so we can give our
users the necessary power to swap arguments by letting them do so in
the spreadsheet (DSL), rather than in code.

Next up we apply mapc to each row, where mapc takes each of the
functions we defined at the top in our function composition DSL and
applies them to specific columns in the spreadsheet.

These functions perform some initial input validation, and convert the
types from strings into more meaningful values [fn:1].

Next we use =derive-column= to apply a function to an existing column
and put the result in a new column at the end of the spreadsheet.
Here we take the postcode and convert it into a URI.

Currently the DSL has explicitly avoided supporting multiple parameter
function calls.  However it is clear that they are needed, so we need
a constrained way to allow the operation to occur.

=fuse= allows just this.  It takes an arbitrary number of column ids,
here the year and month, and applies each column to the supplied
function as an argument.  You could imagine a simple user interface
would easily allow users to select a function and the columns you want
to apply.  This DSL is ideally the only place we would allow the user
to use multi-argument functions.

Next we use derive-colum and fuse to build two more URI's, one for the
facility and the other for the monthly attendances.  Note that we show
the use of a user supplied javascript functions that we defined
earlier.

*** Transform Templates : Triple Templates Revisited

I have developed Bill's triple templates idea, to support a
constrained data-based syntax using clojure vectors:

#+BEGIN_SRC clojure
((graphify [facility-uri name attendance date street-address city postcode website postcode-uri
                    _ observation-uri]

{'fraction_non_alphanumeric': 0.06038496791934005, 'fraction_numerical': 0.009019248395967003, 'mean_word_length': 2.199155524278677, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 13, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 92, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3125660', 'n_tokens_mistral': 7267, 'n_tokens_neox': 6744, 'n_words': 3422}