<story><title>Monitoring demystified: A guide for logging, tracing, metrics</title><url>https://techbeacon.com/enterprise-it/monitoring-demystified-guide-logging-tracing-metrics</url></story><parent_chain></parent_chain><comment><author>buro9</author><text>A lot of excellent information in that blog post and linked from it... but if you&amp;#x27;re wondering where to start:&lt;p&gt;1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don&amp;#x27;t worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number), and that the message will help you debug.&lt;p&gt;2. Instrument metrics using Prometheus, there are libraries that make this easy: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;prometheus.io&amp;#x2F;docs&amp;#x2F;instrumenting&amp;#x2F;clientlibs&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;prometheus.io&amp;#x2F;docs&amp;#x2F;instrumenting&amp;#x2F;clientlibs&amp;#x2F;&lt;/a&gt; . Counts get you started, but you probably want to think in aggregation and to ask about the rate of things and percentiles. Use histograms for this &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;prometheus.io&amp;#x2F;docs&amp;#x2F;practices&amp;#x2F;histograms&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;prometheus.io&amp;#x2F;docs&amp;#x2F;practices&amp;#x2F;histograms&amp;#x2F;&lt;/a&gt; . Use labels to create a more complex picture, i.e. A histogram of HTTP request times with a label of HTTP method means you can see all reqs, just the POST, or maybe the HEAD, GET together, etc... and then create rates over time, percentiles, etc. Do think about cardinality of label values, HTTP methods is good, but request identifiers are bad in high traffic environments... labels should group not identify.&lt;p&gt;Start with those things, tracing follows good logging and metrics as it takes a little more effort to instrument an entire system whereas logging and metrics are valuable even when only small parts of a system are instrumented.&lt;p&gt;Once you&amp;#x27;ve instrumented... Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;grafana.com&amp;#x2F;products&amp;#x2F;cloud&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;grafana.com&amp;#x2F;products&amp;#x2F;cloud&amp;#x2F;&lt;/a&gt; so you can see the results of your work immediately.&lt;p&gt;If it&amp;#x27;s a big project, you have a lot of options and I assume you know them already, this is when you start looking at Cortex and Thanos, Datadog and Loki, tracing with Jaegar.</text></comment>
<story><title>Monitoring demystified: A guide for logging, tracing, metrics</title><url>https://techbeacon.com/enterprise-it/monitoring-demystified-guide-logging-tracing-metrics</url></story><parent_chain></parent_chain><comment><author>KaiserPro</author><text>A few things I have learnt along the way:&lt;p&gt;Logs are great, but only once you&amp;#x27;ve identified the problem. If you are searching through logs to _find_ a problem, its far too late.&lt;p&gt;Processing&amp;#x2F;streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after&amp;#x2F;writing&amp;#x2F;decomming (example: dont use access logs to collect 4xx&amp;#x2F;5xx and make a graph, collate and push the metrics directly)&lt;p&gt;Raw metrics are pretty useless. They need to be manipulated into buisness goals: service x is producing 3% 5xx errors vs % of visitors unable to perform action x&lt;p&gt;Alerts must be actionable.&lt;p&gt;Alerts rules must be based on sensible clear cut rules: service x&amp;#x27;s response time is breeching its SLA not service x&amp;#x27;s response time is double its average for this time in may.</text></comment>