<commit_before>Tucker Sync API
===============

Synchronisation over HTTPS, using POST and JSON.
*Version 0.4*

Motivation
----------

With the advent of multiple device per user mobile computing, synchronising data has become a very common operation. Put simply synchronisation is an expected feature of any application that works with data. While there still appears to be a dearth of complete, clearly explained patterns or sync algorithms in the open source world. This API hopes to provide a better starting point. Two way, three way and more way sync ;-)

Complete with client and server implementation (Python and PHP).

[Tucker Sync API is very much a work in progress, we hope to have a version 1.0 release in due course.]

Terminology
-----------

A **client** is identified uniquely in the server-client relationship.  
A **user** is identified uniquely and may have multiple clients running on different devices.  
A **device** may have multiple users and each is a unique client.  

syncCount - camel case indicates a server value.  
last_sync - underscore indicates a client value.  

Operation
---------

Synchronisation is always initiated by the client application.  
Although it may be prompted to do so by another channel, for example a Cloud Messaging Service.

Sync occurs in two phases:

**Download Phase** - client requests new and remotely changed objects from server since last sync.  
**Upload Phase** - client uploads new and locally changed objects to the server.  

The server maintains a universal sync counter (syncCount).  
This is used in preference to the more fickle timestamp sometimes used in sync API.  

Download Phase
--------------

For each object class:

 - The **client** requests new and changed objects since last_sync:
     - SELECT last_sync FROM object_class ORDER BY last_sync DESC LIMIT 1;
 - The **server** returns objects, for object_class and user where (lastSync > last_sync).
 - For each object returned the **client**:
     - Inserts or updates the object based on server_object_id.
     - Using all of the server provided values and setting local_changes=0.
     - Hence server always wins a.k.a. first-in-first-served or discard type conflict resolution.

Upload Phase
------------

For each object class:

 - The **client** may create objects locally and must set:
     -  server_object_id=0, last_sync=0 and local_changes=1.
 - The **client** may update objects locally and must set:
     -  local_changes=1.
 - The **client** uploads objects to the server (where local_changes=1).
 - The **server** performs an insert on new objects (where server_object_id=0).
 - The **server** performs an update on existing objects:
     - by server_object_id and lastSync < last_sync and client_id is authorised.
     - conflicts are resolved by lastSync, highest winning, since it’s a newer version.
 - The **server** returns an error code and all of the objects.
 - The **client** updates local objects by clientObjectId:
     - with the supplied serverObjectId and lastSync
     - setting local_changes=0

This pattern would generally sync all objects in a class. With an arbitrary maximum data payload of say 5MB. But sync payloads could be chunked by either side if required.

UUIDs are not used to identify objects due to their size impact on the client. Instead the client generates a single UUID to identify itself to the server and sends it’s local id for each object thus allowing the server to identify duplicate new objects. Achieved by setting a unique constraint, see Server Schema.

All requests require an application key to be provided in the query. This is a private key only known to the server and client. If not required the key may be set to an empty string ("") in both the server and client configurations.

An unauthenticated base data download may be performed to pull any base data objects. All other requests require authentication by email and password against an account on the server. The account open request uses the email and password to create and then authenticate an account.

The client should probably perform a background sync on startup and a blocking/modal sync during operation with background on exit. The client may wish to perform a base data download on first start.

**Base URL Example**  
https://api.app.example.com/  
(Use a .htaccess rewrite rule with index.php)  

Test Request
------------

**Summary** - Test the connection and server availability.

**Function** - The test function will perform some basic availability tests on the server and reply with the appropriate error value. The client should inspect the API error code to determine that the server is available and functioning correctly. This call can be useful when first contacting the server, or after a failed communication to check the server before trying again. The matching API error code will be returned to indicate whether authentication succeeded or not and thus both connection and authentication can be checked separately.

**Request**
Query: ?type=test  
Method: POST  

*Example request URL:*

    https://api.app.example.com/?type=test&key=private&email=<EMAIL>&password=<PASSWORD>

*Example request body:*

    {"data":{}}

**Response**  
Message Body: JSON object containing error code.

*Example response code:* 200  
*Example response body:*  

    {"error":1}

Base Data Download Request
--------------------------

**Summary** - Download base data objects for class.

**Function** - A server may provide a set of base data for an object class. This request does not require authentication (email and password). But does require the application key.

**Request**  
Query: ?type=baseDataDown  
Method: POST  
Message Body: JSON object containing objectClass, lastSync and clientUUID.<commit_msg>Improve: minor format corrections in algorithm definition. Markdown trailing spaces.<commit_after>Tucker Sync API
===============

Synchronisation over HTTPS, using POST and JSON.
*Version 0.4*

Motivation
----------

With the advent of multiple device per user mobile computing, synchronising data has become a very common operation. Put simply synchronisation is an expected feature of any application that works with data. While there still appears to be a dearth of complete, clearly explained patterns or sync algorithms in the open source world. This API hopes to provide a better starting point. Two way, three way and more way sync ;-)

Complete with client and server implementation (Python and PHP).

[Tucker Sync API is very much a work in progress, we hope to have a version 1.0 release in due course.]

Terminology
-----------

A **client** is identified uniquely in the server-client relationship.  
A **user** is identified uniquely and may have multiple clients running on different devices.  
A **device** may have multiple users and each is a unique client.  

syncCount - camel case indicates a server value.  
last_sync - underscore indicates a client value.  

Operation
---------

Synchronisation is always initiated by the client application.  
Although it may be prompted to do so by another channel, for example a Cloud Messaging Service.

Sync occurs in two phases:

**Download Phase** - client requests new and remotely changed objects from server since last sync.  
**Upload Phase** - client uploads new and locally changed objects to the server.  

The server maintains a universal sync counter (syncCount).  
This is used in preference to the more fickle timestamp sometimes used in sync API.  

Download Phase
--------------

For each object class:

 - The **client** requests new and changed objects since last_sync:
     - SELECT last_sync FROM object_class ORDER BY last_sync DESC LIMIT 1;
 - The **server** returns objects, for object_class and user where (lastSync > last_sync).
 - For each object returned the **client**:
     - Inserts or updates the object based on server_object_id.
     - Using all of the server provided values and setting local_changes=0.
     - Hence server always wins a.k.a. first-in-first-served or discard type conflict resolution.

Upload Phase
------------

For each object class:

 - The **client** may create objects locally and must set:
     -  server_object_id=0, last_sync=0 and local_changes=1.
 - The **client** may update objects locally and must set:
     -  local_changes=1.
 - The **client** uploads objects to the server (where local_changes=1).
 - The **server** performs an insert on new objects (where server_object_id=0).
 - The **server** performs an update on existing objects:
     - by server_object_id and lastSync < last_sync and client_id is authorised.
     - conflicts are resolved by lastSync, highest winning, since it’s a newer version.
 - The **server** returns an error code and all of the objects.
 - The **client** updates local objects by clientObjectId:
     - with the supplied serverObjectId and lastSync
     - setting local_changes=0

This pattern would generally sync all objects in a class. With an arbitrary maximum data payload of say 5MB. But sync payloads could be chunked by either side if required.

UUIDs are not used to identify objects due to their size impact on the client. Instead the client generates a single UUID to identify itself to the server and sends it’s local id for each object thus allowing the server to identify duplicate new objects. Achieved by setting a unique constraint, see Server Schema.

All requests require an application key to be provided in the query. This is a private key only known to the server and client. If not required the key may be set to an empty string ("") in both the server and client configurations.

An unauthenticated base data download may be performed to pull any base data objects. All other requests require authentication by email and password against an account on the server. The account open request uses the email and password to create and then authenticate an account.

The client should probably perform a background sync on startup and a blocking/modal sync during operation with background on exit. The client may wish to perform a base data download on first start.

**Base URL Example**  
https://api.app.example.com/  
(Use a .htaccess rewrite rule with index.php)  

Test Request
------------

**Summary** - Test the connection and server availability.

**Function** - The test function will perform some basic availability tests on the server and reply with the appropriate error value. The client should inspect the API error code to determine that the server is available and functioning correctly. This call can be useful when first contacting the server, or after a failed communication to check the server before trying again. The matching API error code will be returned to indicate whether authentication succeeded or not and thus both connection and authentication can be checked separately.

**Request**  
Query: ?type=test  
Method: POST  

*Example request URL:*

    https://api.app.example.com/?type=test&key=private&email=<EMAIL>&password=<PASSWORD>

*Example request body:*

    {"data":{}}

**Response**  
Message Body: JSON object containing error code.

*Example response code:* 200  
*Example response body:*  

    {"error":1}

Base Data Download Request
--------------------------

**Summary** - Download base data objects for class.

**Function** - A server may provide a set of base data for an object class. This request does not require authentication (email and password). But does require the application key.

**Request**  
Query: ?type=baseDataDown  
Method: POST  
Message Body: JSON object containing objectClass, lastSync and clientUUID.
{'fraction_non_alphanumeric': 0.0721224103842517, 'fraction_numerical': 0.003094644545688988, 'mean_word_length': 3.7995049504950495, 'pattern_counts': {'":': 4, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3239711', 'n_tokens_mistral': 2781, 'n_tokens_neox': 2715, 'n_words': 1626}