<story><title>Monitoring Is a Pain</title><url>https://matduggan.com/were-all-doing-metrics-wrong/</url></story><parent_chain><item><author>KaiserPro</author><text>The Way(tm) that I was taught&amp;#x2F;experienced is as follows:&lt;p&gt;o Logs are there for ignoring. You need them precisely twice: once when you are developing, and once when the thing&amp;#x27;s gone to shit, but they are never verbose enough when you need them.&lt;p&gt;o Using logs to derive metrics is an expensive fools errand pushed by splunk and the cloud equivalents(ie cloudwatch and the like). Its slow, inaccurate and horrendously expensive.&lt;p&gt;o using logs for monitoring is a fools errand. Its always too slow, and really really fucking brittle.&lt;p&gt;o metrics are king.&lt;p&gt;o pull model metrics is an antipattern&lt;p&gt;o Graphite + grafana is still actually quite good, although time resolution isn&amp;#x27;t there.&lt;p&gt;o You need to raid your metrics stores&lt;p&gt;o We had a bunch of metrics servers in a raid 1, which were then in a raid 0 for performance, all behind loadbalancers and DNS Cnames with a really low TTL.&lt;p&gt;o Cloudwatch metrics are utterly shite&lt;p&gt;o Cloudwatch is actually entirely shit.&lt;p&gt;o tracing is great, and brilliant for performance monitoring.&lt;p&gt;o Xray from AWS is good, but only really for lambdas.&lt;p&gt;o tracing is fragile and doesn&amp;#x27;t really plug and play end to end, unless you have the engineering discipline to enforce &amp;quot;the one true&amp;quot; tracing system &lt;i&gt;everywhere&lt;/i&gt;&lt;p&gt;&lt;i&gt;but what do you monitor?&lt;/i&gt;&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;widgetsandshit.com&amp;#x2F;teddziuba&amp;#x2F;2011&amp;#x2F;03&amp;#x2F;monitoring-theory.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http:&amp;#x2F;&amp;#x2F;widgetsandshit.com&amp;#x2F;teddziuba&amp;#x2F;2011&amp;#x2F;03&amp;#x2F;monitoring-theor...&lt;/a&gt; this still is canonical.&lt;p&gt;In short, everything should have a minimum set of graphs, CPU, Memory, connections, upstream service response times hits per second and query time, at a minimum.&lt;p&gt;You can then aggregate those metrics into a &amp;quot;service health&amp;quot; gauge, where you set a minimum level of service (ie no response time greater than 600ms, and no 5xx&amp;#x2F;4xx errors or similar) red == the service isn&amp;#x27;t performing within spec, yellow == its close to being outside spec, green == its inside spec.&lt;p&gt;if you are running a monolith, then each subsection needs to have a &amp;quot;gauge&amp;quot;. for microservice people, every microservice. You can aggregate  all those gauges into &amp;quot;business services&amp;quot; to make a dashboard that even CEOs can understand.</text></item></parent_chain><comment><author>perlgeek</author><text>I think it hasn&amp;#x27;t really been settled if pushing or pulling metrics is the anti-pattern, it seems to change every 5 to 10 years which one is currently hot.</text></comment>
<story><title>Monitoring Is a Pain</title><url>https://matduggan.com/were-all-doing-metrics-wrong/</url></story><parent_chain><item><author>KaiserPro</author><text>The Way(tm) that I was taught&amp;#x2F;experienced is as follows:&lt;p&gt;o Logs are there for ignoring. You need them precisely twice: once when you are developing, and once when the thing&amp;#x27;s gone to shit, but they are never verbose enough when you need them.&lt;p&gt;o Using logs to derive metrics is an expensive fools errand pushed by splunk and the cloud equivalents(ie cloudwatch and the like). Its slow, inaccurate and horrendously expensive.&lt;p&gt;o using logs for monitoring is a fools errand. Its always too slow, and really really fucking brittle.&lt;p&gt;o metrics are king.&lt;p&gt;o pull model metrics is an antipattern&lt;p&gt;o Graphite + grafana is still actually quite good, although time resolution isn&amp;#x27;t there.&lt;p&gt;o You need to raid your metrics stores&lt;p&gt;o We had a bunch of metrics servers in a raid 1, which were then in a raid 0 for performance, all behind loadbalancers and DNS Cnames with a really low TTL.&lt;p&gt;o Cloudwatch metrics are utterly shite&lt;p&gt;o Cloudwatch is actually entirely shit.&lt;p&gt;o tracing is great, and brilliant for performance monitoring.&lt;p&gt;o Xray from AWS is good, but only really for lambdas.&lt;p&gt;o tracing is fragile and doesn&amp;#x27;t really plug and play end to end, unless you have the engineering discipline to enforce &amp;quot;the one true&amp;quot; tracing system &lt;i&gt;everywhere&lt;/i&gt;&lt;p&gt;&lt;i&gt;but what do you monitor?&lt;/i&gt;&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;widgetsandshit.com&amp;#x2F;teddziuba&amp;#x2F;2011&amp;#x2F;03&amp;#x2F;monitoring-theory.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http:&amp;#x2F;&amp;#x2F;widgetsandshit.com&amp;#x2F;teddziuba&amp;#x2F;2011&amp;#x2F;03&amp;#x2F;monitoring-theor...&lt;/a&gt; this still is canonical.&lt;p&gt;In short, everything should have a minimum set of graphs, CPU, Memory, connections, upstream service response times hits per second and query time, at a minimum.&lt;p&gt;You can then aggregate those metrics into a &amp;quot;service health&amp;quot; gauge, where you set a minimum level of service (ie no response time greater than 600ms, and no 5xx&amp;#x2F;4xx errors or similar) red == the service isn&amp;#x27;t performing within spec, yellow == its close to being outside spec, green == its inside spec.&lt;p&gt;if you are running a monolith, then each subsection needs to have a &amp;quot;gauge&amp;quot;. for microservice people, every microservice. You can aggregate  all those gauges into &amp;quot;business services&amp;quot; to make a dashboard that even CEOs can understand.</text></item></parent_chain><comment><author>pnt12</author><text>I don&amp;#x27;t know a lot about monitoring, why do you say pulling is an anti pattern?&lt;p&gt;My understanding is that with the push pattern you submit a metric when it&amp;#x27;s available, with the pull pattern you make them available via an interface.&lt;p&gt;I&amp;#x27;ve read the first is not as performant, as it leads to submitting lots of metrics. Although I can think of an alternative, which is storing them and pushing batches periodically?</text></comment>