<commit_before>POSTGRES-JSON
===============

## Overview

A simple interface for immutable persistence of Common Lisp objects
(that can be JSON serialized) to a PostgreSQL 9.4+ relation using the
jsonb type.

PostgreSQL 9.4 (in Beta as at late November 2014) is a rock solid
RDBMS which now also supports
[JSON types](http://www.postgresql.org/docs/9.4/static/datatype-json.html).
For many years Marijn Haverbeke's [Postmodern](http://marijnhaverbeke.nl/postmodern/)
has provided an excellent Common Lisp interface to PostgreSQL.

Common Lisp also has some good JSON libraries as compared by Sabra On
The Hill: [JSON libraries
comparison](https://sites.google.com/site/sabraonthehill/home/json-libraries).
[Yason](http://common-lisp.net/project/yason/) is a dependency of this project
but in fact you can use whatever Common Lisp JSON library you like.

Having spent some time working with the full blown NoSQL database
RethinkDB, thanks to the wonderful
[cl-rethinkdb](https://github.com/orthecreedence/cl-rethinkdb), I began
to wonder just how far one might get with the new JSON types in
Postgres.  This library represents a few steps in that direction.

## Status

This library is **under development**. The very simple interface of
the persistence model is probably stable and it compiles and seems to
work, but there is still much to do before an Alpha release.

## Quickstart

You will need a working PostgreSQL 9.4 beta install.  On Debian I
followed the instructions at
https://wiki.postgresql.org/wiki/Apt.  There is a FAQ on
getting the 9.4 beta which you should read to get the apt source line.
Since I already had an earlier install, Debian put the new PostgreSQL
cluster at port 5433 rather than 5432.  YMMV.

Make sure you can quickload Postmodern and connect to some database
with a form like:

`(pomo:connect-toplevel "mydb" "myusername" "" "localhost" :port 5433)`

If you now get a result from `(pomo:query "select 1")` you are ready
to go.

Clone this repo to sit under your `~/quicklisp/local-projects` and do

`(ql:quickload :postgres-json)`.

(Sometimes I need to delete
`~/quicklisp/local-projects/system-index.txt` for this to work).

Now evaluate these forms at the REPL:

```common-lisp
(defpackage :pj-test
  (:use :cl :postgres-json)
  (:shadowing-import-from :postgres-json :get :delete)
  (:import-from :postgres-json :obj :pp-json))

(in-package :pj-test)

;; Once only operation, makes a DB schema for our models
(create-backend)

;; Make a new model to store JSON docs on cats
(create-model 'cat)
```

In the output below I have elided some of the return values for brevity.
`obj` is a trivial function to turn a list of pairs into a hash table.<commit_msg>README fix on once only for create model<commit_after>POSTGRES-JSON
===============

## Overview

A simple interface for immutable persistence of Common Lisp objects
(that can be JSON serialized) to a PostgreSQL 9.4+ relation using the
jsonb type.

PostgreSQL 9.4 (in Beta as at late November 2014) is a rock solid
RDBMS which now also supports
[JSON types](http://www.postgresql.org/docs/9.4/static/datatype-json.html).
For many years M<NAME>ke's [Postmodern](http://marijnhaverbeke.nl/postmodern/)
has provided an excellent Common Lisp interface to PostgreSQL.

Common Lisp also has some good JSON libraries as compared by Sabra On
The Hill: [JSON libraries
comparison](https://sites.google.com/site/sabraonthehill/home/json-libraries).
[Yason](http://common-lisp.net/project/yason/) is a dependency of this project
but in fact you can use whatever Common Lisp JSON library you like.

Having spent some time working with the full blown NoSQL database
RethinkDB, thanks to the wonderful
[cl-rethinkdb](https://github.com/orthecreedence/cl-rethinkdb), I began
to wonder just how far one might get with the new JSON types in
Postgres.  This library represents a few steps in that direction.

## Status

This library is **under development**. The very simple interface of
the persistence model is probably stable and it compiles and seems to
work, but there is still much to do before an Alpha release.

## Quickstart

You will need a working PostgreSQL 9.4 beta install.  On Debian I
followed the instructions at
https://wiki.postgresql.org/wiki/Apt.  There is a FAQ on
getting the 9.4 beta which you should read to get the apt source line.
Since I already had an earlier install, Debian put the new PostgreSQL
cluster at port 5433 rather than 5432.  YMMV.

Make sure you can quickload Postmodern and connect to some database
with a form like:

`(pomo:connect-toplevel "mydb" "myusername" "" "localhost" :port 5433)`

If you now get a result from `(pomo:query "select 1")` you are ready
to go.

Clone this repo to sit under your `~/quicklisp/local-projects` and do

`(ql:quickload :postgres-json)`.

(Sometimes I need to delete
`~/quicklisp/local-projects/system-index.txt` for this to work).

Now evaluate these forms at the REPL:

```common-lisp
(defpackage :pj-test
  (:use :cl :postgres-json)
  (:shadowing-import-from :postgres-json :get :delete)
  (:import-from :postgres-json :obj :pp-json))

(in-package :pj-test)

;; Once only operation, make a DB schema for all our models
(create-backend)

;; Once only operation, make a new model to store JSON docs on cats
(create-model 'cat)
```

In the output below I have elided some of the return values for brevity.
`obj` is a trivial function to turn a list of pairs into a hash table.
{'fraction_non_alphanumeric': 0.08750921149594694, 'fraction_numerical': 0.009948415622697125, 'mean_word_length': 4.391261171797418, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '580673', 'n_tokens_mistral': 1694, 'n_tokens_neox': 1609, 'n_words': 743}