<commit_before>= Developer Guide - Crunchy Proxy
Crunchy Data Solutions, Inc.
v0.0.1, {docdate}
image::crunchy_logo.png?raw=true[]

== Building

Assuming an installation directory of *$HOME/gdev*, please follow the following steps to build *crunchy proxy* from source:
....
mkdir -p $HOME/gdev/src $HOME/gdev/pkg $HOME/gdev/bin
export GOPATH=$HOME/gdev;export GOBIN=$GOPATH/bin;export PATH=$PATH:$GOBIN
....

First, install a golang compiler. As an example, it is possible to:
....
sudo yum -y install golang 
....

Next, pull the source code as follows:
....
cd gdev/src
go get github.com/tools/godep
mkdir github.com/crunchydata
cd github.com/crunchydata
git clone <EMAIL>:CrunchyData/crunchy-proxy.git
....

Next, build the binary as follows:
....
cd crunchy-proxy
godep restore
make
....

== Design
The example shows a message traveling down this path:

*pg client->proxy->pg server->proxy->pg client*

=== Packages

The proxy code is implemented in the following golang packages:

.proxy golang packages
[width="90%",frame="topbot",cols="30,70", options="header"]
|======================
|Package Name | Purpose
|adapter        |adapters can be applied to in-bound and out-bound message flows to do add capabilities like logging or auditing
|admin        |the administration interface, a REST API 
|config        |the configuration file format
|proxy        |the main proxy processing
|testclient        | a test client that uses libpq, useful for testing
|tests        | standalone unit tests
|======================

=== Makefile Targets

The Makefile has the following targets defined:

.Makefile targets
[width="90%",frame="topbot",cols="30,70", options="header"]
|======================
|Makefile Target | Purpose
|gendeps        |calls godep to generate dependencies for golang compilation
|docsbuild        |calls asciidoctor to generate PDF and HTML versions of the documentation
|clean        |removes the proxy binaries 
|dockerimage        |builds the docker image for the proxy
|push        | pushes the docker image to dockerhub
|run        | runs the proxy in foreground using the default configuration file
|test        | executes the standalone unit tests
|======================

=== Connection Pooling

*crunchy proxy* provisions a connection pool for each backend (master and replica(s)) that is defined in the proxy 
configuration file.  The connection pool is a fixed size currently and established before the proxy begins to accept connections from clients.

The connections in the pool are determined by the pool settings found within the configuration parameters *credentials* and *pool*.

Currently *crunchy proxy* only supports basic PostgreSQL password authentication using username and password.

As client requests come into the proxy, the proxy will choose to which backend to route the SQL statement and then pick a free connection from the backend's 
connection pool.

For each connection pool there is a golang channel defined to manage which connections are available for use when processing a SQL statement.  
After the SQL statement is processed, the connection is returned to the pool.  You can think of the pool's channel as a queue of available connections.

=== Client Authentication

Each client must authenticate against the master backend before the proxy will process future client requests.  *crunchy proxy* does not include
an authentication store itself, but instead relies on the master backend to perform authentication.

Once a client does authenticate, the proxy will terminate the client's connection to the master and subsequently begin using the connections 
from the connection pools.

=== Annotations

SQL statements that include a SQL comment of a particular format will be used to determine the routing of a SQL statement either to a master or a replica.<commit_msg>update docs<commit_after>= Developer Guide - Crunchy Proxy
Crunchy Data Solutions, Inc.
v0.0.1, {docdate}
image::crunchy_logo.png?raw=true[]

== Building

Assuming an installation directory of *$HOME/gdev*, please follow the following steps to build *crunchy proxy* from source:
....
mkdir -p $HOME/gdev/src $HOME/gdev/pkg $HOME/gdev/bin
export GOPATH=$HOME/gdev;export GOBIN=$GOPATH/bin;export PATH=$PATH:$GOBIN
....

First, install a golang compiler. As an example, it is possible to:
....
sudo yum -y install golang 
....

Next, pull the source code as follows:
....
cd gdev/src
go get github.com/tools/godep
mkdir github.com/crunchydata
cd github.com/crunchydata
git clone <EMAIL>:CrunchyData/crunchy-proxy.git
....

Next, build the binary as follows:
....
cd crunchy-proxy
godep restore
make
....

== Design
The example shows a message traveling down this path:

*pg client->proxy->pg server->proxy->pg client*

=== Packages

The proxy code is implemented in the following golang packages:

.proxy golang packages
[width="90%",frame="topbot",cols="30,70", options="header"]
|======================
|Package Name | Purpose
|adapter        |adapters can be applied to in-bound and out-bound message flows to do add capabilities like logging or auditing
|admin        |the administration interface, a REST API 
|config        |the configuration file format
|proxy        |the main proxy processing
|testclient        | a test client that uses libpq, useful for testing
|tests        | standalone unit tests
|======================

=== Makefile Targets

The Makefile has the following targets defined:

.Makefile targets
[width="90%",frame="topbot",cols="30,70", options="header"]
|======================
|Makefile Target | Purpose
|gendeps        |calls godep to generate dependencies for golang compilation
|docsbuild        |calls asciidoctor to generate PDF and HTML versions of the documentation
|clean        |removes the proxy binaries 
|dockerimage        |builds the docker image for the proxy
|push        | pushes the docker image to dockerhub
|run        | runs the proxy in foreground using the default configuration file
|test        | executes the standalone unit tests
|======================

=== Wire Protocol

*crunchy-proxy* operates at the PostgreSQL wire protocol (network) layer to understand
PostgreSQL client authentication requests and SQL statements passed
by a client to a PostgreSQL backend.

The proxy does very little processing of the messages sent between
a client and an actual backend, mostly examining the SQL statements
for a proxy-specific annotation.  The annotation is used to route
the message to the backend.

Its important to note that the proxy does not implement all features
of libpq or provide an application interface similar to a JDBC driver
or other language driver.

=== Connection Pooling

*crunchy proxy* provisions a connection pool for each backend (master and replica(s)) that is defined in the proxy 
configuration file.  The connection pool is a fixed size currently and established before the proxy begins to accept connections from clients.

The connections in the pool are determined by the pool settings found within the configuration parameters *credentials* and *pool*.

Currently *crunchy proxy* only supports basic PostgreSQL password authentication using username and password.

As client requests come into the proxy, the proxy will choose to which backend to route the SQL statement and then pick a free connection from the backend's 
connection pool.

For each connection pool there is a golang channel defined to manage which connections are available for use when processing a SQL statement.  
After the SQL statement is processed, the connection is returned to the pool.  You can think of the pool's channel as a queue of available connections.

=== Client Authentication

Each client must authenticate against the master backend before the proxy will process future client requests.  *crunchy proxy* does not include
an authentication store itself, but instead relies on the master backend to perform authentication.

Once a client does authenticate, the proxy will terminate the client's connection to the master and subsequently begin using the connections 
from the connection pools.

=== Annotations

SQL statements that include a SQL comment of a particular format will be used to determine the routing of a SQL statement either to a master or a replica.
{'fraction_non_alphanumeric': 0.08301064859632139, 'fraction_numerical': 0.0036302032913843175, 'mean_word_length': 4.342598577892695, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3088940', 'n_tokens_mistral': 2127, 'n_tokens_neox': 2009, 'n_words': 1067}