<commit_before># Guble Messaging Server

Guble is a simple user-facing messaging and data replication server written in Go.

[![Release](https://img.shields.io/github/release/smancke/guble.svg)](https://github.com/smancke/guble/releases/latest)
[![Docker](https://img.shields.io/docker/pulls/smancke/guble.svg)](https://hub.docker.com/r/smancke/guble/)
[![Build Status](https://api.travis-ci.org/smancke/guble.svg?branch=master)](https://travis-ci.org/smancke/guble)
[![Go Report Card](https://goreportcard.com/badge/github.com/smancke/guble)](https://goreportcard.com/report/github.com/smancke/guble)
[![codebeat badge](https://codebeat.co/badges/7f317892-0a7b-4e31-97f4-a530cf779889)](https://codebeat.co/projects/github-com-smancke-guble)
[![Coverage Status](https://coveralls.io/repos/smancke/guble/badge.svg?branch=master&service=github)](https://coveralls.io/github/smancke/guble?branch=master)
[![GoDoc](https://godoc.org/github.com/smancke/guble?status.svg)](https://godoc.org/github.com/smancke/guble)
[![Awesome-Go](https://camo.githubusercontent.com/13c4e50d88df7178ae1882a203ed57b641674f94/68747470733a2f2f63646e2e7261776769742e636f6d2f73696e647265736f726875732f617765736f6d652f643733303566333864323966656437386661383536353265336136336531353464643865383832392f6d656469612f62616467652e737667)](https://awesome-go.com)

# Overview
Guble is in an early state (release 0.3). 
It is already working well and is very useful, but the protocol, API and storage formats 
may still change (until reaching 0.7). 
If you intend to use guble, please get in contact with us.

The goal of guble is to be a simple and fast message bus for user interaction and replication of data between multiple devices:
* Very easy consumption of messages with web and mobile clients
* Fast realtime messaging, as well as playback of messages from a persistent commit log
* Reliable and scalable over multiple nodes
* User-aware semantics to easily support messaging scenarios between people using multiple devices
* Batteries included: usable as front-facing server, without the need of a proxy layer
* Self-contained: no mandatory dependencies to other services

## Working Features (0.3)

* Publishing and subscription of messages to topics and subtopics
* Persistent message store with transparent live and offline fetching
* WebSocket and REST APIs for message publishing
* Commandline client and Go client library
* Firebase Cloud Messaging (FCM) adapter: delivery of messages as FCM push notifications
* Docker images for server and client
* Simple Authentication and Access-Management
* Clean shutdown
* Improved logging using [logrus](https://github.com/Sirupsen/logrus) and logstash formatter
* Health-Check with Endpoint
* Collection of Basic Metrics, with Endpoint
* Added Postgresql as KV Backend
* Load testing with 5000 messages per instance


## Throughput
Measured on an old notebook with i5-2520M, dual core and SSD. Message payload was 'Hello Word'.
Load driver and server were set up on the same machine, so 50% of the cpu was allocated to the load driver.

* End-2-End: Delivery of ~35.000 persistent messages per second
* Fetching: Receive of ~70.000 persistent messages per second

During the tests, the memory consumption of the server was around ~25 MB.

## Table of Contents

- [Roadmap](#roadmap)
  - [Roadmap Release 0.4](#roadmap-release-04)
  - [Roadmap Release 0.5](#roadmap-release-05)
  - [Roadmap Release 0.6](#roadmap-release-06)
  - [Roadmap Release 0.7](#roadmap-release-07)
- [Guble Docker Image](#guble-docker-image)
  - [Start the Guble Server](#start-the-guble-server)
  - [Connecting with the Guble Client](#connecting-with-the-guble-client)
- [Build and Run](#build-and-run)
  - [Build and Start the Server](#build-and-start-the-server)
  - [Run All Tests](#run-all-tests)
- [Clients](#clients)
- [Protocol Reference](#protocol-reference)
  - [REST API](#rest-api)
    - [Headers](#headers)
  - [WebSocket Protocol](#websocket-protocol)
    - [Message Format](#message-format)
    - [Client Commands](#client-commands)
    - [Server Status Messages](#server-status-messages)
  - [Topics](#topics)
    - [Subtopics](#subtopics)

# Roadmap
This is the current (and fast changing) roadmap and todo list:

## Roadmap Release 0.4
* Support for Apple Push Notification services (a new connector alongside Firebase)
* Upgrade, cleanup, abstraction, documentation, and test coverage of the Firebase connector
* GET list of subscribers / list of topics per subscriber (userID , deviceID) 

## Roadmap Release 0.5
* Support for SMS-sending (a new connector alongside Firebase)
* Replication across multiple servers (in a Guble cluster)
* Acknowledgement of message delivery for connectors
* Storing the sequence-Id of topics in KV store, if we turn off persistence
* Filtering of messages in guble server (e.g. sent by the REST client) according to URL parameters: UserID, DeviceID, Connector name
* Updating README to show subscribe/unsubscribe/get/posting, health/metrics 

## Roadmap Release 0.6
* Make notification messages optional by client configuration
* Correct behaviour of receive command with `maxCount` on subtopics
* Cancel of fetch in the message store and multiple concurrent fetch commands for the same topic
* Configuration of different persistence strategies for topics
* Delivery semantics: user must read on one device / deliver only to one device / notify if not connected, etc.
* User-specific persistent subscriptions across all clients of the user
* Client: (re-)setup of subscriptions after client reconnect
* Message size limit configurable by the client with fetching by URL

## Roadmap Release 0.7
* HTTPS support in the service
* Minimal example: chat application
* Stable JavaScript client: https://github.com/smancke/guble-js
* (TBD) Improved authentication and access-management
* (TBD) Add Consul as KV Backend
* (TBD) Index-based search of messages using [GoLucene](https://github.com/balzaczyy/golucene)

# Guble Docker Image<commit_msg>Release 0.4: updated README<commit_after># Guble Messaging Server

Guble is a simple user-facing messaging and data replication server written in Go.

[![Release](https://img.shields.io/github/release/smancke/guble.svg)](https://github.com/smancke/guble/releases/latest)
[![Docker](https://img.shields.io/docker/pulls/smancke/guble.svg)](https://hub.docker.com/r/smancke/guble/)
[![Build Status](https://api.travis-ci.org/smancke/guble.svg?branch=master)](https://travis-ci.org/smancke/guble)
[![Go Report Card](https://goreportcard.com/badge/github.com/smancke/guble)](https://goreportcard.com/report/github.com/smancke/guble)
[![codebeat badge](https://codebeat.co/badges/7f317892-0a7b-4e31-97f4-a530cf779889)](https://codebeat.co/projects/github-com-smancke-guble)
[![Coverage Status](https://coveralls.io/repos/smancke/guble/badge.svg?branch=master&service=github)](https://coveralls.io/github/smancke/guble?branch=master)
[![GoDoc](https://godoc.org/github.com/smancke/guble?status.svg)](https://godoc.org/github.com/smancke/guble)
[![Awesome-Go](https://camo.githubusercontent.com/13c4e50d88df7178ae1882a203ed57b641674f94/68747470733a2f2f63646e2e7261776769742e636f6d2f73696e647265736f726875732f617765736f6d652f643733303566333864323966656437386661383536353265336136336531353464643865383832392f6d656469612f62616467652e737667)](https://awesome-go.com)

# Overview
Guble is in an early state (release 0.4). 
It is already working well and is very useful, but the protocol, API and storage formats 
may still change (until reaching 0.7). 
If you intend to use guble, please get in contact with us.

The goal of guble is to be a simple and fast message bus for user interaction and replication of data between multiple devices:
* Very easy consumption of messages with web and mobile clients
* Fast realtime messaging, as well as playback of messages from a persistent commit log
* Reliable and scalable over multiple nodes
* User-aware semantics to easily support messaging scenarios between people using multiple devices
* Batteries included: usable as front-facing server, without the need of a proxy layer
* Self-contained: no mandatory dependencies to other services

## Working Features (0.4)

* Publishing and subscription of messages to topics and subtopics
* Persistent message store with transparent live and offline fetching
* WebSocket and REST APIs for message publishing
* Commandline client and Go client library
* Firebase Cloud Messaging (FCM) adapter: delivery of messages as FCM push notifications
* Docker images for server and client
* Simple Authentication and Access-Management
* Clean shutdown
* Improved logging using [logrus](https://github.com/Sirupsen/logrus) and logstash formatter
* Health-Check with Endpoint
* Collection of Basic Metrics, with Endpoint
* Added Postgresql as KV Backend
* Load testing with 5000 messages per instance
* Support for Apple Push Notification services (a new connector alongside Firebase)
* Upgrade, cleanup, abstraction, documentation, and test coverage of the Firebase connector
* GET list of subscribers / list of topics per subscriber (userID , deviceID) 
* Support for SMS-sending using Nexmo (a new connector alongside Firebase)

## Throughput
Measured on an old notebook with i5-2520M, dual core and SSD. Message payload was 'Hello Word'.
Load driver and server were set up on the same machine, so 50% of the cpu was allocated to the load driver.

* End-2-End: Delivery of ~35.000 persistent messages per second
* Fetching: Receive of ~70.000 persistent messages per second

During the tests, the memory consumption of the server was around ~25 MB.

## Table of Contents

- [Roadmap](#roadmap)
  - [Roadmap Release 0.5](#roadmap-release-05)
  - [Roadmap Release 0.6](#roadmap-release-06)
  - [Roadmap Release 0.7](#roadmap-release-07)
- [Guble Docker Image](#guble-docker-image)
  - [Start the Guble Server](#start-the-guble-server)
  - [Connecting with the Guble Client](#connecting-with-the-guble-client)
- [Build and Run](#build-and-run)
  - [Build and Start the Server](#build-and-start-the-server)
  - [Run All Tests](#run-all-tests)
- [Clients](#clients)
- [Protocol Reference](#protocol-reference)
  - [REST API](#rest-api)
    - [Headers](#headers)
  - [WebSocket Protocol](#websocket-protocol)
    - [Message Format](#message-format)
    - [Client Commands](#client-commands)
    - [Server Status Messages](#server-status-messages)
  - [Topics](#topics)
    - [Subtopics](#subtopics)

# Roadmap
This is the current (and fast changing) roadmap and todo list:

## Roadmap Release 0.5
* Replication across multiple servers (in a Guble cluster)
* Acknowledgement of message delivery for connectors
* Storing the sequence-Id of topics in KV store, if we turn off persistence
* Filtering of messages in guble server (e.g. sent by the REST client) according to URL parameters: UserID, DeviceID, Connector name
* Updating README to show subscribe/unsubscribe/get/posting, health/metrics 

## Roadmap Release 0.6
* Make notification messages optional by client configuration
* Correct behaviour of receive command with `maxCount` on subtopics
* Cancel of fetch in the message store and multiple concurrent fetch commands for the same topic
* Configuration of different persistence strategies for topics
* Delivery semantics: user must read on one device / deliver only to one device / notify if not connected, etc.
* User-specific persistent subscriptions across all clients of the user
* Client: (re-)setup of subscriptions after client reconnect
* Message size limit configurable by the client with fetching by URL

## Roadmap Release 0.7
* HTTPS support in the service
* Minimal example: chat application
* Stable JavaScript client: https://github.com/smancke/guble-js
* (TBD) Improved authentication and access-management
* (TBD) Add Consul as KV Backend
* (TBD) Index-based search of messages using [GoLucene](https://github.com/balzaczyy/golucene)

# Guble Docker Image
{'fraction_non_alphanumeric': 0.09639564124056998, 'fraction_numerical': 0.04777870913663034, 'mean_word_length': 5.105936540429887, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 38, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5761971', 'n_tokens_mistral': 4062, 'n_tokens_neox': 3504, 'n_words': 1206}