<story><title>We need new data books, so we started one: Cloud Data Management</title><url>https://chartio.com/blog/cloud-data-management-book-launch/</url></story><parent_chain><item><author>jameslk</author><text>One of the experiences that stuck with me most going from doing software engineering stuff to data engineering stuff is the very difficult and sometimes complete lack of tooling around testing and debugging issues in SQL and pipelines. As software engineers, we have debuggers, unit testing, mocking frameworks, e2e tools, BDD languages like Cucumber, code quality tools, etc. But when you&amp;#x27;re working on a pipeline and you&amp;#x27;re wondering what will happen if you run it, sometimes your best bet is actually to just run it and wait 20 min for it to complete. Or run some portion of it ad hoc. Or if you want to know what a SQL query is going to do in the black box of your query engine, you might try to parse the esoteric language of a query explainer. The best tools seem to be available after deployment such as data quality tests, dashboards and alerts. I think there&amp;#x27;s a lot of opportunity to improve the ecosystem.</text></item></parent_chain><comment><author>chrisweekly</author><text>Agreed. Outside the world of ML (but working with reasonably large data, ie a few billion rows of server log entries in varying formats) I had some success grabbing a narrow slice (single-digit millions) and experimenting w&amp;#x2F; ways to query it using combination of regex and SQL, thanks to an amazing little powertool called lnav (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;lnav.org&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;lnav.org&lt;/a&gt;). It&amp;#x27;s like an intuitive, scriptable, mini-ETL on the commandline you can learn in no time. I once used it to track down the root cause of a p0 in prod, in parallel to the ops gurus and most senior devs using dedicated tools and processes; we arrived at the same conclusion around the same time, and the stuff I&amp;#x27;d extracted helped speed the patch. Felt pretty good, and I&amp;#x27;ve been singing lnav&amp;#x27;s praises ever since.&lt;p&gt;Shoot I must be getting old, rambling on a tangent. Get off my lawn! ;)</text></comment>
<story><title>We need new data books, so we started one: Cloud Data Management</title><url>https://chartio.com/blog/cloud-data-management-book-launch/</url></story><parent_chain><item><author>jameslk</author><text>One of the experiences that stuck with me most going from doing software engineering stuff to data engineering stuff is the very difficult and sometimes complete lack of tooling around testing and debugging issues in SQL and pipelines. As software engineers, we have debuggers, unit testing, mocking frameworks, e2e tools, BDD languages like Cucumber, code quality tools, etc. But when you&amp;#x27;re working on a pipeline and you&amp;#x27;re wondering what will happen if you run it, sometimes your best bet is actually to just run it and wait 20 min for it to complete. Or run some portion of it ad hoc. Or if you want to know what a SQL query is going to do in the black box of your query engine, you might try to parse the esoteric language of a query explainer. The best tools seem to be available after deployment such as data quality tests, dashboards and alerts. I think there&amp;#x27;s a lot of opportunity to improve the ecosystem.</text></item></parent_chain><comment><author>heroHACK17</author><text>Writing this comment as I &amp;#x27;actually run it and wait 20 min for it to complete.&amp;#x27;</text></comment>