<story><title>Show HN: Log collector that runs on a $4 VPS</title><url>https://github.com/Nevin1901/erlog</url><text>Hey guys, I&amp;#x27;m building erlog to try and solve problems with logging. While trying to add logs to my application, I couldn&amp;#x27;t find any lightweight log platform which was easy to set up without adding tons of dependencies to my code, or configuring 10,000 files.&lt;p&gt;ErLog is just a simple go web server which batch inserts json logs into an sqlite3 server. Through tuning sqlite3 and batching inserts, I find I can get around 8k log insertions&amp;#x2F;sec which is fast enough for small projects.&lt;p&gt;This is just an MVP, and I plan to add more features once I talk to users. If anyone has any problems with logging, feel free to leave a comment and I&amp;#x27;d love to help you out.</text></story><parent_chain></parent_chain><comment><author>Dachande663</author><text>I’ve found the hard part is not so much the collection of logs (especially at this scale), but the eventual querying. If you’ve got an unknown set of fields been logged, queries very quickly devolve into lots of slow table scans or needing materialised views that start hampering your ingest rate.&lt;p&gt;I settled on a happy&amp;#x2F;ok midpoint recently whereby I dump logs in a redis queue using filebeat as it’s very simple. Then have a really simple queue consumer that dumps the logs into clickhouse using a schema Uber detailed (split keys and values), so queries can be pretty quick even over arbitrary fields. 30,00 logs an hour and I can normally search for anything in under a second.</text></comment>
<story><title>Show HN: Log collector that runs on a $4 VPS</title><url>https://github.com/Nevin1901/erlog</url><text>Hey guys, I&amp;#x27;m building erlog to try and solve problems with logging. While trying to add logs to my application, I couldn&amp;#x27;t find any lightweight log platform which was easy to set up without adding tons of dependencies to my code, or configuring 10,000 files.&lt;p&gt;ErLog is just a simple go web server which batch inserts json logs into an sqlite3 server. Through tuning sqlite3 and batching inserts, I find I can get around 8k log insertions&amp;#x2F;sec which is fast enough for small projects.&lt;p&gt;This is just an MVP, and I plan to add more features once I talk to users. If anyone has any problems with logging, feel free to leave a comment and I&amp;#x27;d love to help you out.</text></story><parent_chain></parent_chain><comment><author>unxdfa</author><text>I see your idea but you could drop the JSON and use rsyslogd + logrotate + grep? You can grep 10 gig files on a $5 VPS easily and quickly! I can&amp;#x27;t speak for a $4 one ;)</text></comment>