<commit_before>Jenerator: a user-defined machine learning system
=================================================

Problem description
-------------------

Jubatus is going to be a framework to implement `online machine learning which scales automatically`.
But currently it has only a set of direct implementations of classifier, regression and recommender.
And there are 9 x N declarations or implemenations for eash learning machine's API, where N is a number
of API. For example, classifier currently has 7 APIs (train, classify, set_config, get_config, save,
 load and get_status). Thus there're 9 x 7 = 63 reputations of function declarations or implementations.
Recommender has 12 APIs therefore we'd 84 reputations. (M variety of client libraries will increase
 the number: (M + 8) x N )

A developer of a new machine learning algorithm would have seen a misery of Do Repeat Yourself.


Moreover, composable feature conversion and machine learning must be available in a near future.
With direct implementation of C++ language it is very difficult to minimize the cost of user-
implementation. Deciding a specification of Domain Specific Language will enable us a
 under-the-wood optimization and interface improvement.


There are several preliminary condition - first of all, for all developers C++ should be available
because they have many existing software implementation of there exellent ML algorithms.
Secondly, the new system must have high usability to write, compile, deploy, test and use it. The
more difficult it is, the less developers will come.


Solution
========

Simple msgpack-idl parser and a bit complex C++ code generator. Input specification of IDL is
very simple and a extension of msgpack-idl. Users should annotate 3 decorators on each RPC-API
of each services, which are about routing, R/W feature and aggregator.

routing
 - cht
 - broadcast
 - random

R/W feature
 - update   - this does changes the server state, guarded by writer lock.
 - analysis - does not change the server state, so that threads can work in parallel.

 

Users can use the input jubatus-IDL file. Output is a set of C++ files that is to be assembled
to the Jubatus servers: main server-kickstart program, keeper server to be the gateway and C++
server template (different from C++ 'template').

- Input: 'xxx.idl' which implements msgpack-idl based extended spec
- Output0: 'xxx_impl.cpp', 'xxx_server.hpp' which implements developer's application in server-side including 'main()'
- Output1: 'xxx_keeper.cpp' -- keeper implementation includes 'main()'
- Output2: 'xxx_client.hpp' client implementation
- Output3: 'xxx_types.hpp' type definitions which are used in server, keeper and client
- Output4: optional: 'xxx_serv.tmpl.(h|c)pp' -- server template for users


Licence of generated codes
==========================

Jenerator itself is distributed under LGPL, but you can distribute all generated codes under terms of your choice.


Reference
=========

- msgpack

 - http://msgpack.org

- msgpack-idl spec

 - https://github.com/msgpack/msgpack-haskell/blob/master/msgpack-idl/Specification.md

Things left
===========

- parse version number in service
- other file inclusion
- comment parsing ('#\n' cannot be parsed)
- stop using polimorphic variant<commit_msg>Fix rst style of jenerator's readme<commit_after>Jenerator: a user-defined machine learning system
=================================================

Problem description
-------------------

Jubatus is going to be a framework to implement `online machine learning which scales automatically`.
But currently it has only a set of direct implementations of classifier, regression and recommender.
And there are 9 x N declarations or implemenations for eash learning machine's API, where N is a number
of API. For example, classifier currently has 7 APIs (train, classify, set_config, get_config, save,
load and get_status). Thus there're 9 x 7 = 63 reputations of function declarations or implementations.
Recommender has 12 APIs therefore we'd 84 reputations. (M variety of client libraries will increase
the number: (M + 8) x N )

A developer of a new machine learning algorithm would have seen a misery of Do Repeat Yourself.


Moreover, composable feature conversion and machine learning must be available in a near future.
With direct implementation of C++ language it is very difficult to minimize the cost of user-
implementation. Deciding a specification of Domain Specific Language will enable us a
under-the-wood optimization and interface improvement.


There are several preliminary condition - first of all, for all developers C++ should be available
because they have many existing software implementation of there exellent ML algorithms.
Secondly, the new system must have high usability to write, compile, deploy, test and use it. The
more difficult it is, the less developers will come.


Solution
--------

Simple msgpack-idl parser and a bit complex C++ code generator. Input specification of IDL is
very simple and a extension of msgpack-idl. Users should annotate 3 decorators on each RPC-API
of each services, which are about routing, R/W feature and aggregator.

routing
  - cht
  - broadcast
  - random

R/W feature
  - update   - this does changes the server state, guarded by writer lock.
  - analysis - does not change the server state, so that threads can work in parallel.

 

Users can use the input jubatus-IDL file. Output is a set of C++ files that is to be assembled
to the Jubatus servers: main server-kickstart program, keeper server to be the gateway and C++
server template (different from C++ 'template').

- Input: 'xxx.idl' which implements msgpack-idl based extended spec
- Output0: 'xxx_impl.cpp', 'xxx_server.hpp' which implements developer's application in server-side including 'main()'
- Output1: 'xxx_keeper.cpp' -- keeper implementation includes 'main()'
- Output2: 'xxx_client.hpp' client implementation
- Output3: 'xxx_types.hpp' type definitions which are used in server, keeper and client
- Output4: optional: 'xxx_serv.tmpl.(h|c)pp' -- server template for users


Licence of generated codes
--------------------------

Jenerator itself is distributed under LGPL, but you can distribute all generated codes under terms of your choice.


Reference
---------

- msgpack

  - http://msgpack.org

- msgpack-idl spec

  - https://github.com/msgpack/msgpack-haskell/blob/master/msgpack-idl/Specification.md

Things left
-----------

- parse version number in service
- other file inclusion
- comment parsing ('#\n' cannot be parsed)
- stop using polimorphic variant
{'fraction_non_alphanumeric': 0.0926881067961165, 'fraction_numerical': 0.0051577669902912624, 'mean_word_length': 4.142745709828393, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2854626', 'n_tokens_mistral': 1739, 'n_tokens_neox': 1609, 'n_words': 872}