<story><title>GPT-4 is getting worse over time, not better</title><url>https://twitter.com/svpino/status/1681614284613099520</url></story><parent_chain><item><author>phillipcarter</author><text>The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things.&lt;p&gt;This topic has come up before, and my hypothesis is still that GPT-4 hasn&amp;#x27;t gotten worse, it&amp;#x27;s just that the magic has worn off as we&amp;#x27;ve used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.</text></item></parent_chain><comment><author>r3trohack3r</author><text>I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge.&lt;p&gt;The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back then, ported dirbuster to POSIX compliant multi-threaded C by name only. It required three prompts, roughly:&lt;p&gt;* “port dirbuster to POSIX compliant c”&lt;p&gt;* “that’s great! You’re almost there, but wordlists are not prefixed with a &amp;#x2F;, you’ll need to add that yourself. Can you generate a diff updating the file with the fix?”&lt;p&gt;* “This is pretty slow, can we make it more aggressive at scanning?”&lt;p&gt;It helped me write a daemon for FreeBSD that accepted jails definitions as a declarative manifest and managed them in a diff-reconciliation loop kubernetes style. It implemented a binpacking algorithm to fit an arbitrary number of randomly sized images onto a single sheet of A1 paper.&lt;p&gt;Most programming tasks I threw at it, it could work its way through with a little guidance if it could fit in a prompt.&lt;p&gt;Now, it’s basically worthless at helping me with programming tasks beyond trivial problems.&lt;p&gt;Folks who had early access before it was public have commented on how exceptional it was back then. But also how terrifyingly unaligned it was. And the more they aligned the model with acceptable social behavior, the worse the model performed. Alignment goals like “don’t help people plan mass killings” seem to cause regressions in the models performance.&lt;p&gt;I wouldn’t dismiss these comments. If they’re true, it means there is a hyper intelligent early GPT-4 model sitting on a HDD somewhere that dwarfs what we’ve seen publicly. A poorly aligned model that’s down to help no matter what your request is.</text></comment>
<story><title>GPT-4 is getting worse over time, not better</title><url>https://twitter.com/svpino/status/1681614284613099520</url></story><parent_chain><item><author>phillipcarter</author><text>The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things.&lt;p&gt;This topic has come up before, and my hypothesis is still that GPT-4 hasn&amp;#x27;t gotten worse, it&amp;#x27;s just that the magic has worn off as we&amp;#x27;ve used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.</text></item></parent_chain><comment><author>majkinetor</author><text>So the original paper from Stanford and Berkley is also linked to this AI influencer? I am really amazed by this kind of dismissal. Its totally irrelevant who posted the info and how framed it is, as long as you have access to the source.</text></comment>