<story><title>BlackRock shelves unexplainable AI liquidity models</title><url>https://www.risk.net/asset-management/6119616/blackrock-shelves-unexplainable-ai-liquidity-models</url></story><parent_chain></parent_chain><comment><author>michaelbuckbee</author><text>I learned a new term in the context of AI recently: &amp;quot;Specification Gaming&amp;quot;.&lt;p&gt;There&amp;#x27;s a big list here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;t.co&amp;#x2F;OqoYN8MvMN&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;t.co&amp;#x2F;OqoYN8MvMN&lt;/a&gt;&lt;p&gt;But it&amp;#x27;s stuff like:&lt;p&gt;- Evolved algorithm for landing aircraft exploited overflow errors in the physics simulator by creating large forces that were estimated to be zero, resulting in a perfect score&lt;p&gt;- A cooperative GAN architecture for converting images from one genre to another (eg horses&amp;lt;-&amp;gt;zebras) has a loss function that rewards accurate reconstruction of images from its transformed version; CycleGAN turns out to partially solve the task by, in addition to the cross-domain analogies it learns, steganographically hiding autoencoder-style data about the original image invisibly inside the transformed image to assist the reconstruction of details.&lt;p&gt;- Simulated pancake making robot learned to throw the pancake as high in the air as possible in order to maximize time away from the ground&lt;p&gt;- Robot hand pretending to grasp an object by moving between the camera and the object&lt;p&gt;- Self-driving car rewarded for speed learns to spin in circles&lt;p&gt;All of which leads me to think that if you can&amp;#x27;t at some level explain how&amp;#x2F;what&amp;#x2F;why it&amp;#x27;s reaching a certain conclusion that it may be reaching a radically different end than you&amp;#x27;re anticipating.</text></comment>
<story><title>BlackRock shelves unexplainable AI liquidity models</title><url>https://www.risk.net/asset-management/6119616/blackrock-shelves-unexplainable-ai-liquidity-models</url></story><parent_chain></parent_chain><comment><author>agentofoblivion</author><text>I hear this a lot. In my opinion, people overestimate their ability to “understand” non-neural net models.&lt;p&gt;For instance, take the go-to classification model:  Logistic Regression. Many people think they can draw insight by looking at the coefficients on the variables. If it’s 2.0 for variable A and 1.0 for variable B, then A must move the needle twice as much.&lt;p&gt;But not so fast. B, for instance, might be correlated with A. In this case, the coefficients are also correlated and interpretability becomes much more nuanced. And this isn’t the exception, it’s the rule. If you have a lot of features, chances are many of them are correlated.&lt;p&gt;In addition, your variables likely operate at different scales, so you’ll have needed to normalize and scale everything, which makes another layer of abstraction between you and interpretation. This becomes even more complicated when you consider encoded categorical variables. Are you trying to interpret each category independently, or assess their importance as a group?  Not obvious how to make these aggregations. The story only gets more complicated for e.g. Random Forests.&lt;p&gt;I think it’s best to accept that you can’t interpret these models very well in general. At least in the case of some models (like neural nets), they approximate a Bayesian posterior, which has some nice properties.</text></comment>