Q:

Prometheus - devise custom metrics for the average order-size, order-processing-time and time between orders

Working with Prometheus metrics requires a different way of thinking. Devising metrics is often not too difficult. Can you help me devise metrics for an order processing system? 
A system processes requests (like orders). Each request contains a number of items to be processed. The processing time for a request may differ between a few seconds to hours. The number of items in 1 request varies between 1 and a million. There may be some time between the requests. 
How can I devise metrics (counter, gauge, histogram, summary?) with Prometheus that provides info during the day (say every 10 minutes) on: 

Average number of items (in a request) processed per time-unit
Average processing time of a request
Average (waiting) time between requests

In the application per request the following information is available: start-time, end-time, number-of-items-processed. 

A:

The most useful Prometheus metric is a counter. It is basically a forever-increasing value for the lifetime of the process (that starts again from zero whenever the monitored process restarts). Based on it, you can compute how much the monitored metric has increased over any time range (the previous 5 minutes, yesterday, year-to-date) simply by subtracting its value at the start of the range from its value at the end. (And adjusting for any resets to zero.) Prometheus does this automatically for you via the increase() and rate() functions it provides.
For your specific use case, the average number of items processed per time unit would be e.g. rate(items_processed_total[5m]), where rate() is provided by Prometheus; items_processed_total is a counter you will have to define and increment every time you process an item (or a batch of them); and 5m is the time range you want the average to be computed over (you can obviously use an arbitrary value, as long as it's a few times longer than your scrape interval). This would give you the QPS average, i.e. the number of items processed per second, averaged over 5 minutes.
For your second bullet point (average processing time of a request) you'll need two counters, say requests_processed_total (incremented by 1 every time you finish processing a request) and request_processing_time_seconds_total (that you increment by request_end_time - request_start_time every time you finish processing a request). The value you're looking for would then be produced by
rate(request_processing_time_seconds_total[5m]) / rate(requests_processed_total[5m])

I.e. how much the request processing time increased over the previous 5 minutes divided by the number of requests processed over the previous 5 minutes. (Or an arbitrary time range of your choice.)
For your final bullet point (average waiting time between requests), that's either
1 / rate(requests_processed_total[5m])

(i.e. the inverse of "requests per second", i.e. "seconds per request") if you want the average time between requests.
Or, if you're interested in the idle time between requests:
(1 - rate(request_processing_time_seconds_total[5m]))
  /
rate(requests_processed_total[5m])

if you're only interested in the idle time between requests. To explain that, 1 - rate(request_processing_time_seconds_total[5m]) is the percentage of time your job is idle (100% - processing time); and dividing it by the QPS is the same as multiplying by the average interval between requests (see above).
The latter expression assumes this is a single-threaded process, that's either processing a request or else idle. If you are actually processing requests in parallel, "idle time" doesn't make much sense as a metric.
And finally, in case you're interested in more than averages (say the median or other percentiles) histograms are great. If they didn't use an order of magnitude more metrics (and storage and CPU, etc.) I'd use them for everything. E.g. a histogram of request processing times would give you the average processing time, but also the estimated median processing time, and whatever other percentile yuu want (estimated). Plus of course, the number of requests and the total processing time (that come built in). And they allow you to do aggregate metrics across multiple instances, as opposed to a summary metric.

