<story><title>Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup</title><url>https://github.com/Dicklesworthstone/automatic_log_collector_and_analyzer</url><text>My immediate reaction to today&amp;#x27;s news that Splunk was being acquired was to comment in the HN discussion for that story:&lt;p&gt;&amp;quot;I hated Splunk so much that I spent a couple days a few months ago writing a single 1200 line python script that does absolutely everything I need in terms of automatic log collection, ingestion, and analysis from a fleet of cloud instances. It pulls in all the log lines, enriches them with useful metadata like the IP address of the instance, the machine name, the log source, the datetime, etc. and stores it all in SQlite, which it then exposes to a very convenient web interface using Datasette.&lt;p&gt;I put it in a cronjob and it&amp;#x27;s infinitely better (at least for my purposes) than Splunk, which is just a total nightmare to use, and can be customized super easily and quickly. My coworkers all prefer it to Splunk as well. And oh yeah, it&amp;#x27;s totally free instead of costing my company thousands of dollars a year! If I owned CSCO stock I would sell it-- this deal shows incredibly bad judgment.&amp;quot;&lt;p&gt;I had been meaning to clean it up a bit and open-source it but never got around to it. However, someone asked today in response to my comment if I had released it, so I figured now would be a good time to go through it and clean it up, move the constants to an .env file, and create a README.&lt;p&gt;This code is obviously tailored to my own requirements for my project, but if you know Python, it&amp;#x27;s extremely straightforward to customize it for your own logs (plus, some of the logs are generic, like systemd logs, and the output of netstat&amp;#x2F;ss&amp;#x2F;lsof, which it combines to get a table of open connections by process over time for each machine-- extremely useful for finding code that is leaking connections!). And I also included the actual sample log files from my project that correspond to the parsing functions in the code, so you can easily reason by analogy to adapt it to your own log files.&lt;p&gt;As many people pointed out in responses to my comment, this is obviously not a real replacement for Splunk for enterprise users who are ingesting terabytes a day from thousands of machines and hundreds of sources. If it were, hopefully someone would be paying me $28 billion for it instead of me giving it away for free! But if you don&amp;#x27;t have a huge number of machines and really hate using Splunk while wasting thousands of dollars, this might be for you.</text></story><parent_chain><item><author>rollcat</author><text>&amp;quot;This simple tool solves X at my org&amp;quot; is probably the most underrated type of project. There&amp;#x27;s not enough room to overcomplicate something that isn&amp;#x27;t a core part of the business, it must be practical to maintain, simple&amp;amp;stupid enough so that onboarding is not a hurdle, etc.&lt;p&gt;I encourage everyone to share your &amp;quot;splunk in 1kloc of Python&amp;quot; projects! Some of my own:&lt;p&gt;- &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;judo&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;judo&lt;/a&gt; is Ansible without Python or YAML&lt;p&gt;- &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;zfs-autosnap&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;zfs-autosnap&lt;/a&gt; manages rolling ZFS snapshots</text></item></parent_chain><comment><author>eigenvalue</author><text>Thanks, based on the dismissive replies to my original comment in the Splunk acquisition discussion, I thought this would get a lot of hostile takes saying that it was dumb, that I reinvented the wheel because I didn&amp;#x27;t want to spend 2 weeks trying to figure out opentelemetry nonsense and tools X, Y, and Z, that it was trivial, that it wouldn&amp;#x27;t scale, etc.&lt;p&gt;But people are actually being surprisingly nice and friendly! I guess people just really hate Splunk!</text></comment>
<story><title>Show HN: My Single-File Python Script I Used to Replace Splunk in My Startup</title><url>https://github.com/Dicklesworthstone/automatic_log_collector_and_analyzer</url><text>My immediate reaction to today&amp;#x27;s news that Splunk was being acquired was to comment in the HN discussion for that story:&lt;p&gt;&amp;quot;I hated Splunk so much that I spent a couple days a few months ago writing a single 1200 line python script that does absolutely everything I need in terms of automatic log collection, ingestion, and analysis from a fleet of cloud instances. It pulls in all the log lines, enriches them with useful metadata like the IP address of the instance, the machine name, the log source, the datetime, etc. and stores it all in SQlite, which it then exposes to a very convenient web interface using Datasette.&lt;p&gt;I put it in a cronjob and it&amp;#x27;s infinitely better (at least for my purposes) than Splunk, which is just a total nightmare to use, and can be customized super easily and quickly. My coworkers all prefer it to Splunk as well. And oh yeah, it&amp;#x27;s totally free instead of costing my company thousands of dollars a year! If I owned CSCO stock I would sell it-- this deal shows incredibly bad judgment.&amp;quot;&lt;p&gt;I had been meaning to clean it up a bit and open-source it but never got around to it. However, someone asked today in response to my comment if I had released it, so I figured now would be a good time to go through it and clean it up, move the constants to an .env file, and create a README.&lt;p&gt;This code is obviously tailored to my own requirements for my project, but if you know Python, it&amp;#x27;s extremely straightforward to customize it for your own logs (plus, some of the logs are generic, like systemd logs, and the output of netstat&amp;#x2F;ss&amp;#x2F;lsof, which it combines to get a table of open connections by process over time for each machine-- extremely useful for finding code that is leaking connections!). And I also included the actual sample log files from my project that correspond to the parsing functions in the code, so you can easily reason by analogy to adapt it to your own log files.&lt;p&gt;As many people pointed out in responses to my comment, this is obviously not a real replacement for Splunk for enterprise users who are ingesting terabytes a day from thousands of machines and hundreds of sources. If it were, hopefully someone would be paying me $28 billion for it instead of me giving it away for free! But if you don&amp;#x27;t have a huge number of machines and really hate using Splunk while wasting thousands of dollars, this might be for you.</text></story><parent_chain><item><author>rollcat</author><text>&amp;quot;This simple tool solves X at my org&amp;quot; is probably the most underrated type of project. There&amp;#x27;s not enough room to overcomplicate something that isn&amp;#x27;t a core part of the business, it must be practical to maintain, simple&amp;amp;stupid enough so that onboarding is not a hurdle, etc.&lt;p&gt;I encourage everyone to share your &amp;quot;splunk in 1kloc of Python&amp;quot; projects! Some of my own:&lt;p&gt;- &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;judo&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;judo&lt;/a&gt; is Ansible without Python or YAML&lt;p&gt;- &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;zfs-autosnap&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rollcat&amp;#x2F;zfs-autosnap&lt;/a&gt; manages rolling ZFS snapshots</text></item></parent_chain><comment><author>aaviator42</author><text>My org&amp;#x27;s apps heavily use this simple key-value interface built on sqlite: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;aaviator42&amp;#x2F;StorX&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;aaviator42&amp;#x2F;StorX&lt;/a&gt;&lt;p&gt;There&amp;#x27;s also a bunch of other purpose-built tiny utilities on that GitHub account: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;aaviator42?tab=repositories&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;aaviator42?tab=repositories&lt;/a&gt;</text></comment>