<commit_before>Devastator
==========

[![Build Status](https://travis-ci.org/nbusy/devastator.svg?branch=master)](https://travis-ci.org/nbusy/devastator) [![GoDoc](https://godoc.org/github.com/nbusy/devastator?status.svg)](https://godoc.org/github.com/nbusy/devastator)

Devastator is a messaging server for delivering chat messages to mobile and Web clients. For each delivery target, the server uses different protocol. i.e. GCM for Android apps, WebSockets for browsers, etc. The server is completely written in Go and makes huge use of goroutines and channels.

Tech Stack
----------

GCM CCS (for message delivery and retrieval from Android clients), GAE Sockets API (CCS XMPP delivery protocol, used in place of plain TCP on AppEngine)

Architecture
------------

Messaging server utilizes device specific delivery infrastructure for notifications and messages; GCM + TLS for Android, APNS + TLS for iOS, and WebSockets for the Web browsers.

```
+-----------+------------+-------------+
| GCM + TLS | APNS + TLS | Web Sockets |
+-----------+------------+-------------+
|           Messaging Server           |
+--------------------------------------+
```

Following is the overview of the server application's components:

```
+---------------------------------------+
|              main + config            |
+---------------------------------------+
|                 server                |
+---------------------------------------+
|   router  |   listener  |  msg queue  |
+---------------------------------------+
```

Client-Server Protocol
----------------------

Client server communication protocol is based on [JSON RPC](http://www.jsonrpc.org/specification) 2.0 specs. Mobile devices connect with the TLS endpoint and the Web browsers utilizes the WebSocket endpoint. The message framing on the TLS endpoint is quite simple:

```
[uint32] 4 Bytes Content Length Header + [JSON] Message Body
```

Following is a valid TLS message frame for connecting mobile devices (except the header, which should be a 4 byte binary encoded uint32):

```
xxxx{method: "ping"}
```

Client Authentication
---------------------

First-time registration is done through Google+ OAuth 2.0 flow. After a successful registration, the connecting device receives a client-side TLS certificate (for mobile devices) or a JSON Web Token (for browsers), to be used for successive connections.

Typical Client-Server Sequence
------------------------------

Client-server communication sequence is pretty similar to that of XMPP, except we are using JSON RPC packaging.

```
[Server]                    [Client]
+                                  +
|---------GCM Notification-------->| [offline]
|                                  |
|                                  |
|<-----------auth.cert-------------| [online]
|                                  |
|----------ACK/batch[msg]--------->|
|                                  |
|<-----------batch[ACK]------------|
|                                  |
|                                  |
|<------------msg.echo-------------|
|                                  |
|---------------ACK--------------->|
|                                  |
|-------------msg.echo------------>|
|                                  |
|<--------------ACK----------------|
+                                  +
```

Any message that was not acknowledged will be delivered to the client again (hence at-least-once delivery princinple), unless TTL of the message was reached.

Testing
-------

All the tests can be executed by `go test -v -race -cover ./...` command. Optionally you can add `-v` flag to observe all connection logs. Integration tests require environment variables defined in the next section. If they are missing, integration tests are skipped.

Environment Variables
---------------------

Following environment variables needs to be present on any dev or production environment:

```bash
export GOOGLE_API_KEY=
export GOOGLE_PREPROD_API_KEY=
```

Performance Notes
-----------------

The messaging server is designed to make max usage of available CPU resources. However exceeding 100% CPU usage will cause a memory usage spike as marshalled/unmarshalled messages and other allocated byte buffers will have to reside in memory much longer. Ideally, server process' CPU usage should never exceed 95% of overall system CPU resources. Currently there is no clustering support but it is a top priority.

License
-------<commit_msg>halt on first race error<commit_after>Devastator
==========

[![Build Status](https://travis-ci.org/nbusy/devastator.svg?branch=master)](https://travis-ci.org/nbusy/devastator) [![GoDoc](https://godoc.org/github.com/nbusy/devastator?status.svg)](https://godoc.org/github.com/nbusy/devastator)

Devastator is a messaging server for delivering chat messages to mobile and Web clients. For each delivery target, the server uses different protocol. i.e. GCM for Android apps, WebSockets for browsers, etc. The server is completely written in Go and makes huge use of goroutines and channels.

Tech Stack
----------

GCM CCS (for message delivery and retrieval from Android clients), GAE Sockets API (CCS XMPP delivery protocol, used in place of plain TCP on AppEngine)

Architecture
------------

Messaging server utilizes device specific delivery infrastructure for notifications and messages; GCM + TLS for Android, APNS + TLS for iOS, and WebSockets for the Web browsers.

```
+-----------+------------+-------------+
| GCM + TLS | APNS + TLS | Web Sockets |
+-----------+------------+-------------+
|           Messaging Server           |
+--------------------------------------+
```

Following is the overview of the server application's components:

```
+---------------------------------------+
|              main + config            |
+---------------------------------------+
|                 server                |
+---------------------------------------+
|   router  |   listener  |  msg queue  |
+---------------------------------------+
```

Client-Server Protocol
----------------------

Client server communication protocol is based on [JSON RPC](http://www.jsonrpc.org/specification) 2.0 specs. Mobile devices connect with the TLS endpoint and the Web browsers utilizes the WebSocket endpoint. The message framing on the TLS endpoint is quite simple:

```
[uint32] 4 Bytes Content Length Header + [JSON] Message Body
```

Following is a valid TLS message frame for connecting mobile devices (except the header, which should be a 4 byte binary encoded uint32):

```
xxxx{method: "ping"}
```

Client Authentication
---------------------

First-time registration is done through Google+ OAuth 2.0 flow. After a successful registration, the connecting device receives a client-side TLS certificate (for mobile devices) or a JSON Web Token (for browsers), to be used for successive connections.

Typical Client-Server Sequence
------------------------------

Client-server communication sequence is pretty similar to that of XMPP, except we are using JSON RPC packaging.

```
[Server]                    [Client]
+                                  +
|---------GCM Notification-------->| [offline]
|                                  |
|                                  |
|<-----------auth.cert-------------| [online]
|                                  |
|----------ACK/batch[msg]--------->|
|                                  |
|<-----------batch[ACK]------------|
|                                  |
|                                  |
|<------------msg.echo-------------|
|                                  |
|---------------ACK--------------->|
|                                  |
|-------------msg.echo------------>|
|                                  |
|<--------------ACK----------------|
+                                  +
```

Any message that was not acknowledged will be delivered to the client again (hence at-least-once delivery princinple), unless TTL of the message was reached.

Testing
-------

All the tests can be executed by `GORACE="halt_on_error=1" go test -v -race -cover ./...` command. Optionally you can add `-v` flag to observe all connection logs. Integration tests require environment variables defined in the next section. If they are missing, integration tests are skipped.

Environment Variables
---------------------

Following environment variables needs to be present on any dev or production environment:

```bash
export GOOGLE_API_KEY=
export GOOGLE_PREPROD_API_KEY=
```

Performance Notes
-----------------

The messaging server is designed to make max usage of available CPU resources. However exceeding 100% CPU usage will cause a memory usage spike as marshalled/unmarshalled messages and other allocated byte buffers will have to reside in memory much longer. Ideally, server process' CPU usage should never exceed 95% of overall system CPU resources. Currently there is no clustering support but it is a top priority.

License
-------
{'fraction_non_alphanumeric': 0.20357541899441342, 'fraction_numerical': 0.003463687150837989, 'mean_word_length': 1.6687537268932617, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2382583', 'n_tokens_mistral': 2157, 'n_tokens_neox': 1958, 'n_words': 861}