<story><title>Monitoring your logs is mostly a tarpit</title><url>https://utcc.utoronto.ca/~cks/space/blog/sysadmin/LogMonitoringTarpit</url></story><parent_chain></parent_chain><comment><author>sethammons</author><text>Structured logging is my jam. Our unit tests validate error content, errors have all dynamic text in different fields, provide all info to reproduce (including, where appropriate, copy-pastable curl commands to replicate an error to remote api endpoints!) and log aggregators like Splunk can group and aggregate and find relationships.&lt;p&gt;If I see a spike in errors, I can, in real time, count them by user id, group by region, filter out by some other field, then have the table of events generate a live line chart easily showing the error rates over time, and pin pointing the exact moment an error started.&lt;p&gt;Less than five to ten minutes after detection or report and I can be talking with another team about a change they released at time $x, including a list of affected users for product teams to reach out to if needed.</text></comment>
<story><title>Monitoring your logs is mostly a tarpit</title><url>https://utcc.utoronto.ca/~cks/space/blog/sysadmin/LogMonitoringTarpit</url></story><parent_chain></parent_chain><comment><author>dale_glass</author><text>Most logging is insanity in action. Behold:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    1.2.3.4 - - [08&amp;#x2F;Aug&amp;#x2F;2023:12:48:11 +0200] &amp;quot;GET &amp;#x2F;wp-config.php.bak HTTP&amp;#x2F;1.1&amp;quot; 404 196 &amp;quot;-&amp;quot; &amp;quot;Mozilla&amp;#x2F;5.0 (Windows NT 6.1; Win64; x64) AppleWebKit&amp;#x2F;537.36 (KHTML, like Gecko) Chrome&amp;#x2F;47.0.2503.0 Safari&amp;#x2F;537.36&amp;quot;

&lt;/code&gt;&lt;/pre&gt;
1. Apache has a perfectly coherent and structured view of the situation: Host 1.2.3.4 made a GET request of this particular URL, with this particular UA, at this particular date.&lt;p&gt;2. Apache proceeds to shove all this into a string, destroying the structure, and probably some of the data (it likely knows the timestamp with &amp;gt;1s precision)&lt;p&gt;3. Then to do some sort of analysis we need to manually reverse this process: write some convoluted regex to match this kind of line, turn an IP address back into a number, turn a timestamp into a number, find the delimiters. We&amp;#x27;re now undoing the damage Apache did. Why!?&lt;p&gt;4. For extra fun this adds extra problems to deal with log rotation, log compression, newlines and special characters in fields, and so on. All kinds of problems that are ultimately unproductive to solve and get in the way of actual useful work.&lt;p&gt;Logging should be structured by nature. I shouldn&amp;#x27;t be writing weird regexps. I should have every message structured from the start, where I can search by &amp;quot;ip_address is in 1.2.3&amp;#x2F;24&amp;quot;, or &amp;quot;request == &amp;#x27;GET&amp;#x27;&amp;quot;, or such. Every bit of data should be logged in its pristine form with zero ambiguity, with its original type, and analyzable accordingly.&lt;p&gt;I think systemd has the beginning of a good idea in here, in that you can actually do this if you care to, and can send arbitrary chunks of data (even binary) to the log if needed.&lt;p&gt;People who complain about journald not logging in plaintext honestly baffle me, because come on, whose idea of fun is it to do log parsing? And why are we spending CPU cycles on converting timestamps to text and parsing them back?</text></comment>