<story><title>The four pillars of data observability: metrics, metadata, lineage, and logs</title><url>https://www.metaplane.dev/blog/the-four-pillars-of-data-observability</url></story><parent_chain><item><author>pbowyer</author><text>We&amp;#x27;re good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs?&lt;p&gt;For example an image processing pipeline. You don&amp;#x27;t always want to log (it&amp;#x27;d never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You&amp;#x27;ve already got text logging for each step, recording metrics like duration and which filters were involved. I have saved files and referenced them in the logfile, but no log viewers I&amp;#x27;ve seen understand anything beyond text. So I then have to build my own UI or open the images in turn.&lt;p&gt;Is there a pattern to handle this?</text></item></parent_chain><comment><author>kzh_</author><text>+1 on existing log viewers being particularly well suited for text over non-textual assets. My experience here is limited but I believe Grafana has a dynamic image plugin if you store a link to an asset in blob storage or Base64 encode it.&lt;p&gt;I&amp;#x27;ve also heard of people storing those links in a database like Snowflake then creating displays on top using Tableau or Looker, to avoid having to build a web app from scratch.</text></comment>
<story><title>The four pillars of data observability: metrics, metadata, lineage, and logs</title><url>https://www.metaplane.dev/blog/the-four-pillars-of-data-observability</url></story><parent_chain><item><author>pbowyer</author><text>We&amp;#x27;re good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs?&lt;p&gt;For example an image processing pipeline. You don&amp;#x27;t always want to log (it&amp;#x27;d never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You&amp;#x27;ve already got text logging for each step, recording metrics like duration and which filters were involved. I have saved files and referenced them in the logfile, but no log viewers I&amp;#x27;ve seen understand anything beyond text. So I then have to build my own UI or open the images in turn.&lt;p&gt;Is there a pattern to handle this?</text></item></parent_chain><comment><author>nerdponx</author><text>It might be interesting to have something like &amp;quot;statistical logging&amp;quot;, which saves the intermediate image files 1% of the time and discards them after 30 days.</text></comment>