<commit_before>Incoming!!
==========

Large file uploads with web browsers are frustrating because you can not implement them in a straightforward and painless way. When implemented wrong, they stall your web app and might even break it, and implementing them right is not easily done. Incoming!! handles large file uploads for your web apps so you don't have to. In the browser, it chops up a large file into little pieces, transfers those over to its own server application, which puts them together, stores them, and finally hands the uploaded file over to your web app backend. Disconnects during upload are no problem, and explicit pause / resume is also supported. With Incoming!!, both the complexity and performance impact of large file uploads are off your web app's back.

Incoming!! consists of a server application and a JavaScript client library. The server can run alongside your web app backend or centrally in your organization, and the JavaScript client is used directly in your web app's frontend in the browser. When you want to upload a large file, your web app backend first fetches an upload ticket from the Incoming!! server. Then, using that ticket, your frontend can use the Incoming!! JavaScript library to send the large file to the Incoming!! server. When the upload is finished, the Incoming!! server hands the file over to your web app backend.

Your web app backend and the Incoming!! server communicate through simple HTTP requests: Incoming!! provides an endpoint for handing out upload tickets, and issues a request to your web app backend when an upload is finished. To its JavaScript library, Incoming!! exposes a WebSocket server. Incoming!! also hosts the JavaScript library file.

The Incoming!! server is implemented in Go, and the JavaScript library in, well, JavaScript, without use of external libraries. The whole thing is free and open source, licensed under the permissive TODO LICENSE.


Status
------

Incoming!! is in development. Most of it is in place and works. It can already be used, but we don't consider it and the API stable yet.

At the present stage of development, Incoming!! can already be deployed together with an individual application, or centrally in your organization. However, for the latter case to be viable, Incoming!! should be able to scale out with several server instances in order to provide enough upload bandwidth. This was always in the backs of our heads during design and implementation, and is the next major feature we will implement.


Usage example
-------------

Simple backend code in Python (using the simple [Bottle](http://bottlepy.org/) web framework) can look like the following.

Request an upload ticket from the Incoming!! server like this (using the excellent and simple [Requests](http://python-requests.org) library):

```python
req_params = { "signalFinishURL" : "http://APP_HOSTNAME/api/backend/upload_finished" }
req = requests.post("http://INCOMING_HOSTNAME/incoming/backend/new_upload",
                     params=req_params)
```
<commit_msg>documentation<commit_after>Incoming!!
==========

Large file uploads with web browsers are frustrating because you can not implement them in a straightforward and painless way. When implemented wrong, they stall your web app and might even break it, and implementing them right is not easily done. Incoming!! handles large file uploads for your web apps so you don't have to. In the browser, it chops up a large file into little pieces, transfers those over to its own server application, which puts them together, stores them, and finally hands the uploaded file over to your web app backend. Disconnects during upload are no problem, and explicit pause / resume is also supported. With Incoming!!, both the complexity and performance impact of large file uploads are off your web app's back.

Incoming!! consists of a server application and a JavaScript client library. The server can run alongside your web app backend or centrally in your organization, and the JavaScript client is used directly in your web app's frontend in the browser.

When you want to upload a large file, your web app backend first fetches an upload ticket from the Incoming!! server. Then, using that ticket, your frontend can use the Incoming!! JavaScript library to send the large file to the Incoming!! server. When the upload is finished, the Incoming!! server hands the file over to your web app backend.

Your web app backend and the Incoming!! server communicate through simple HTTP requests: Incoming!! provides an endpoint for handing out upload tickets, and issues a request to your web app backend when an upload is finished. To its JavaScript library, Incoming!! exposes a WebSocket server. Incoming!! also hosts the JavaScript library file.

The Incoming!! server is implemented in Go, and the JavaScript library in, well, JavaScript, without use of external libraries. The whole thing is free and open source, licensed under the permissive TODO LICENSE.


Status
------

Incoming!! is in development. Most of it is in place and works. It can already be used, but we don't consider it and the API stable yet.

At the present stage of development, Incoming!! can already be deployed together with an individual application, or centrally in your organization. However, for the latter case to be viable, Incoming!! should be able to scale out with several server instances in order to provide ample upload bandwidth. This has always been in the backs of our heads during design and implementation, and is the next major feature we will implement.


Usage example
-------------

Simple backend code in Python (using the simple [Bottle](http://bottlepy.org/) web framework) can look like the following.

Request an upload ticket from the Incoming!! server like this (using the excellent and simple [Requests](http://python-requests.org) library):

```python
req_params = { "signalFinishURL" : "http://APP_HOSTNAME/api/backend/upload_finished" }
req = requests.post("http://INCOMING_HOSTNAME/incoming/backend/new_upload",
                     params=req_params)
```

{'fraction_non_alphanumeric': 0.05871740200098409, 'fraction_numerical': 0.0, 'mean_word_length': 4.488748874887489, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2382932', 'n_tokens_mistral': 1400, 'n_tokens_neox': 1351, 'n_words': 897}