<story><title>Show HN: Metabase, an open-source business intelligence tool</title><url>http://www.metabase.com/</url></story><parent_chain><item><author>jtmarmon</author><text>Sort of off topic, but every cool BI tool I can find (prompted by recently seeing metabase and AWS quicksight) seems to work with one database at a time.&lt;p&gt;When working with microservices, your data is spread across some N databases, rendering most BI tools, from what I can see, completely useless for reporting on more than one service at a time.&lt;p&gt;Are there any solutions for this? Or is the only one right now just dumping all your service DBs into a single DB for analytics?&lt;p&gt;edit: thanks friends. ETL it is. don&amp;#x27;t know why I thought it would make sense to have a tool that reports across multiple databases since the performance would be horrific...although maybe there&amp;#x27;s space for a hosted BI tool that does ETL automagically. just a thought</text></item></parent_chain><comment><author>vog</author><text>You will have N databases no matter what. Almost every company has some legacy systems around, or simply uses different tools for different jobs.&lt;p&gt;However, you still want to copy &amp;amp; combine your data into a single database. The relevant Fowler patterns are:&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;martinfowler.com&amp;#x2F;bliki&amp;#x2F;DataLake.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;martinfowler.com&amp;#x2F;bliki&amp;#x2F;DataLake.html&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;martinfowler.com&amp;#x2F;bliki&amp;#x2F;ReportingDatabase.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;martinfowler.com&amp;#x2F;bliki&amp;#x2F;ReportingDatabase.html&lt;/a&gt;&lt;p&gt;More precicely, it is a common pattern to split the analtics into three parts: 1) Collecting the data (using lots of adapters) into a &amp;quot;data lake&amp;quot;, 2) filter and preprocess that data lake into a uniform data structure whose structure is determined by the analysis goal rather than the operational system, 3) analyze that uniform data with statistical and other tools.&lt;p&gt;It usually makes no sense to combine those phases into a single overall tool. First, these are very different tasks, where different specialized tools will evolve anyway. Second, you want to keep the intermediate results anyway - for caching as well as to have an audit trail and reproducibility of the results.&lt;p&gt;For example, you don&amp;#x27;t want the performance of operational system be affected by for how many analysis tools it is used at a point in point. Also, you don&amp;#x27;t to work on a always-modifying dataset while fine-tunung your analysis methods.</text></comment>
<story><title>Show HN: Metabase, an open-source business intelligence tool</title><url>http://www.metabase.com/</url></story><parent_chain><item><author>jtmarmon</author><text>Sort of off topic, but every cool BI tool I can find (prompted by recently seeing metabase and AWS quicksight) seems to work with one database at a time.&lt;p&gt;When working with microservices, your data is spread across some N databases, rendering most BI tools, from what I can see, completely useless for reporting on more than one service at a time.&lt;p&gt;Are there any solutions for this? Or is the only one right now just dumping all your service DBs into a single DB for analytics?&lt;p&gt;edit: thanks friends. ETL it is. don&amp;#x27;t know why I thought it would make sense to have a tool that reports across multiple databases since the performance would be horrific...although maybe there&amp;#x27;s space for a hosted BI tool that does ETL automagically. just a thought</text></item></parent_chain><comment><author>exelius</author><text>This is why data warehousing exists. That way you can use many different BI tools against a consistent set of data and pre-calculate a lot of commonly-used summary data.&lt;p&gt;ETL also isn&amp;#x27;t something you can just do automagically. It requires an understanding of the data and your goals, because you essentially have to build both your data model and your reporting requirements into your ETL process. You could probably do it automagically for some simple cases, but for most real-world scenarios it&amp;#x27;s just going to be easier to write a Python&amp;#x2F;Perl script to run your ETL for you.&lt;p&gt;BI &amp;#x2F; reporting requires a lot of plumbing to work correctly. You have to set up read-only clones of your DBs for reporting (because you don&amp;#x27;t want to be running large queries against production servers) and generally an ETL process that dumps everything into a data warehouse. From there, you can push subsets of that data out to various BI tools that provide the interface.</text></comment>