<story><title>Automated Unit Test Improvement Using Large Language Models at Meta</title><url>https://arxiv.org/abs/2402.09171</url></story><parent_chain><item><author>tivert</author><text>&amp;gt; 75% of TestGen-LLM&amp;#x27;s test cases built correctly, 57% passed reliably, and 25% increased coverage.&lt;p&gt;The problem I have with LLM generated tests is that it seems highly likely that they&amp;#x27;d &amp;quot;ratify&amp;quot; buggy behavior, and I&amp;#x27;d think that&amp;#x27;d be especially likely if the code-base already had low test coverage.  One of the nice things about writing new tests by hand is you&amp;#x27;ve got someone who can judge if it&amp;#x27;s the system being stupid or if it&amp;#x27;s the test.&lt;p&gt;At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.</text></item></parent_chain><comment><author>ithkuil</author><text>Writing tests is indeed a great opportunity for finding bugs.&lt;p&gt;But a codebase with good test coverage allows you to safely perform large scale refactorings without having regressions and that&amp;#x27;s useful property even if you have bugs and the refactoring preserves them faithfully.&lt;p&gt;The risk of using a tool that generates tests designed to encode the current behaviour is that you may be lulled in a false sense of safety, while all you&amp;#x27;ve done is to encode the current behaviour, as advertised.&lt;p&gt;Perhaps this problem can be just solved by not calling these tests &amp;quot;tests&amp;quot; but something like &amp;quot;behavioural snapshots&amp;quot; or something like that (cannot think of a better name, but the idea is to capture the idea that they were not meant to encode necessarily the correct behaviour but just the current behaviour)</text></comment>
<story><title>Automated Unit Test Improvement Using Large Language Models at Meta</title><url>https://arxiv.org/abs/2402.09171</url></story><parent_chain><item><author>tivert</author><text>&amp;gt; 75% of TestGen-LLM&amp;#x27;s test cases built correctly, 57% passed reliably, and 25% increased coverage.&lt;p&gt;The problem I have with LLM generated tests is that it seems highly likely that they&amp;#x27;d &amp;quot;ratify&amp;quot; buggy behavior, and I&amp;#x27;d think that&amp;#x27;d be especially likely if the code-base already had low test coverage.  One of the nice things about writing new tests by hand is you&amp;#x27;ve got someone who can judge if it&amp;#x27;s the system being stupid or if it&amp;#x27;s the test.&lt;p&gt;At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.</text></item></parent_chain><comment><author>js8</author><text>&amp;gt;  One of the nice things about writing new tests by hand is you&amp;#x27;ve got someone who can judge if it&amp;#x27;s the system being stupid or if it&amp;#x27;s the test.&lt;p&gt;This is an instance of a more general problem, which I call &amp;quot;the problem of unwanted change&amp;quot;. If you have an automated system which can change itself, how do you know a change is actually intended&amp;#x2F;correct or merely a symptom of a bug, failure or imperfect knowledge the automation has?&lt;p&gt;That&amp;#x27;s why I think human supervision is always needed to an extent, to determine what scenario has occured.&lt;p&gt;This happens in all sorts of systems. And people tend to think they can solve this with just another layer of automation like here. Testing was originally invented as a way to check if the program works correctly. If you automate it, you will face the same problem, just with a bigger code (in the form of tests rather than assertions).</text></comment>