<commit_before>
## Tools TODO

- Timing Heroku dyno boot time
- Test whether processing times are normally distributed.
Visual+[analytical test](https://en.wikipedia.org/wiki/Normality_test).
- Calculate clusters/bins to separate multi-mode, non-normal data into
- A runnable+introspectable model of the scaling algorithm. Ability to test it on real/historical data.
- For a given configuration, estimate what loads can (and cannot) be handled

## Simulation

Discrete-event simulation

* [Introduction to Discrete-Event Simulation and the SimPy Language](http://heather.cs.ucdavis.edu/~matloff/156/PLN/DESimIntro.pdf)
* [An introduction to Discrete-Event Simulation](https://www.cs.cmu.edu/~music/cmsip/readings/intro-discrete-event-sim.html),
pedagocical introduction of concepts and possible ways to implement.
* [Sim.js](https://github.com/btelles/simjs-updated), JavaScript library for discrete-event simulation. Updated in 2016, but not much used.

## Queueing theory
https://en.wikipedia.org/wiki/Queueing_theory

> The average delay any given packet is likely to experience is given by the formula 1/(μ-λ)
> where μ is the number of packets per second the facility can sustain
> and λ is the average rate at which packets are arriving to be serviced
https://en.wikipedia.org/wiki/Queuing_delay (networking theory)

> The long-term average number of customers in a stable system L is equal to
> the long-term average effective arrival rate, λ, multiplied by the (Palm‑)average time a customer spends in the system, W;
> or expressed algebraically: L = λW
> 
> "not influenced by the arrival process distribution, the service distribution, the service order, or practically anything else"
https://en.wikipedia.org/wiki/Little%27s_law

<commit_msg>doc: Some notes around simulation<commit_after>
## Tools TODO

- Timing Heroku dyno boot time
- Test whether processing times are normally distributed.
Visual+[analytical test](https://en.wikipedia.org/wiki/Normality_test).
- Calculate clusters/bins to separate multi-mode, non-normal data into
- A runnable+introspectable model of the scaling algorithm. Ability to test it on real/historical data.
- For a given configuration, estimate what loads can (and cannot) be handled

## Simulation

Discrete-event simulation

* [Introduction to Discrete-Event Simulation and the SimPy Language](http://heather.cs.ucdavis.edu/~matloff/156/PLN/DESimIntro.pdf)
* [An introduction to Discrete-Event Simulation](https://www.cs.cmu.edu/~music/cmsip/readings/intro-discrete-event-sim.html),
pedagocical introduction of concepts and possible ways to implement.
* [Sim.js](https://github.com/btelles/simjs-updated), JavaScript library for discrete-event simulation. Updated in 2016, but not much used.

### Aspects

#### Input process
Part of app / execution environment.
Type of distribution. Poisson, uniform
Distribution parameters
Low-load conditions
High-load conditions
Periodic changes
Gradual changes, abrupt changes

#### Job processing
Each assumed to be mostly independent.
However not entirely true. Sharing CPU resources, sharing RAM, sharing disk I/O, network I/O,
at least with other jobs in same worker. Possibly also with other workers, and
even other workers from other apps co-hosted on the same hardware/hypervisor.

#### Worker
Part of app.

Actions: Scale up, scale down. Complete new message.
Spawn time (Cloud service side), startup time (app side).
Shutdown time (app side)

Prefetch setting

Worker crash (and restart)
Worker crashed state (not restarting)
Worker deadlocked. Not accepting new messages

#### Broker service (RabbitMQ)

Interfaces:
HTTP GET for Autoscaling communication
AMQP broker for worker/app communication.

Broker crash/restart
Broker not accepting messages. Quota full etc.
Timeout getting queue info
Unexpected error getting queue info

#### Cloud service (Heroku)

Interface: HTTP GET/POST

Exceeding max workers
Permission denied. Wrong credentials etc
Unexpected error.

#### Autoscaling service (guv)

Role configuration
Scaling algorithm


### General considerations

Many of the aspects considered apply to MsgFlo, without autoscaling.
Also in other systems, like IoT using MQTT/Mosquitto.

Need a componenent models. Using NoFlo.js would be nice for that.
Would be nice to use FBP graphs for setting up network.
But if event-driven the evaluation model does not match, events need
to be executed based on the time the new event is for.

All cross-service comunication goes over network.
There is a non-trivial amount of latency, and a non-zero error rate.
The simulation should be able to model this, including bad and error-conditions.

### Use for end-to-end testing

If simulation can talk the regular protocols, it can be used to run the real service against in end-to-end tests.
Such tests likely need less simulation fidelity, and run with simulation speed == real-time.
But possibly using some data recorded from the past, either from simulation or real system.
Such data can probably be re-dated. 

In end-to-end, should probably mostly be using a standard AMQP broker.
The ability to simulate AMQP broker makes it possible.

## Queueing theory
https://en.wikipedia.org/wiki/Queueing_theory

> The average delay any given packet is likely to experience is given by the formula 1/(μ-λ)
> where μ is the number of packets per second the facility can sustain
> and λ is the average rate at which packets are arriving to be serviced
https://en.wikipedia.org/wiki/Queuing_delay (networking theory)

> The long-term average number of customers in a stable system L is equal to
> the long-term average effective arrival rate, λ, multiplied by the (Palm‑)average time a customer spends in the system, W;
> or expressed algebraically: L = λW
> 
> "not influenced by the arrival process distribution, the service distribution, the service order, or practically anything else"
https://en.wikipedia.org/wiki/Little%27s_law


{'fraction_non_alphanumeric': 0.07286559594251901, 'fraction_numerical': 0.0033812341504649195, 'mean_word_length': 4.688461538461539, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 19, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3168544', 'n_tokens_mistral': 1635, 'n_tokens_neox': 1557, 'n_words': 736}