<story><title>Show HN: GPT-JSON – Structured and typehinted GPT responses in Python</title><url>https://github.com/piercefreeman/gpt-json</url><text>Hey HN, I&amp;#x27;ve been using GPT a lot lately in some side projects around data generation and benchmarking. During the course of prompt tuning I ended up with a pretty complicated request: the value that I was looking for, an explanation, a criticism, etc. JSON was the most natural output format for this but results would often be broken, have wrong types, or contain missing fields.&lt;p&gt;There&amp;#x27;s been some positive movement in this space, like with jsonformer (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1rgs&amp;#x2F;jsonformer&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1rgs&amp;#x2F;jsonformer&lt;/a&gt;) the other day. But nothing that was plug and play with GPT.&lt;p&gt;This library consolidates the separate logic that I built across 5 different projects. It lets you prompt the model for how it should return fields, inject variable prompts, handle common formatting errors, then cast to pydantic when you&amp;#x27;re done for typehinting and validation in your IDE. If you&amp;#x27;re able to play around with it, let me know what you think.</text></story><parent_chain><item><author>crazyedgar</author><text>I like the idea, but I think a library that focuses on producing requests and parsing responses according to schema is better. Sending requests to the server is orthogonal to the purpose.&lt;p&gt;What we&amp;#x27;ve found useful in practice in dealing with similar problems:&lt;p&gt;- Use json5 instead of json when parsing. It allows trailing commas.&lt;p&gt;- Don&amp;#x27;t let it respond in true&amp;#x2F;false. Instead, ask it for a short sentence explaining whether it is true or false. Afterwards, use a small embedding model such as sbert to extract true&amp;#x2F;false from the sentence. We&amp;#x27;ve found that GPT is able to reason better in this case, and it is much more robust.&lt;p&gt;- For numerical scores, do a similar thing by asking GPT for a description, then with the small embedding model write a few examples matching your score scale, and for each response use the score of the best matched example. If you let GPT give you scores directly without explanation, 20% of the time it will give you nonsense.</text></item></parent_chain><comment><author>IanCal</author><text>&amp;gt; Don&amp;#x27;t let it respond in true&amp;#x2F;false. Instead, ask it for a short sentence explaining whether it is true or false. Afterwards, use a small embedding model such as sbert to extract true&amp;#x2F;false from the sentence. We&amp;#x27;ve found that GPT is able to reason better in this case, and it is much more robust.&lt;p&gt;Have you tried just getting it to do both? It reasons far better given some space to think, so I often have it explain things first then give the answer. You&amp;#x27;re effectively then using gpt for the extraction too.&lt;p&gt;This hugely improved the class hierarchies it was creating for me, significantly improving the reuse of classes and using better classes for fields too.</text></comment>
<story><title>Show HN: GPT-JSON – Structured and typehinted GPT responses in Python</title><url>https://github.com/piercefreeman/gpt-json</url><text>Hey HN, I&amp;#x27;ve been using GPT a lot lately in some side projects around data generation and benchmarking. During the course of prompt tuning I ended up with a pretty complicated request: the value that I was looking for, an explanation, a criticism, etc. JSON was the most natural output format for this but results would often be broken, have wrong types, or contain missing fields.&lt;p&gt;There&amp;#x27;s been some positive movement in this space, like with jsonformer (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1rgs&amp;#x2F;jsonformer&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1rgs&amp;#x2F;jsonformer&lt;/a&gt;) the other day. But nothing that was plug and play with GPT.&lt;p&gt;This library consolidates the separate logic that I built across 5 different projects. It lets you prompt the model for how it should return fields, inject variable prompts, handle common formatting errors, then cast to pydantic when you&amp;#x27;re done for typehinting and validation in your IDE. If you&amp;#x27;re able to play around with it, let me know what you think.</text></story><parent_chain><item><author>crazyedgar</author><text>I like the idea, but I think a library that focuses on producing requests and parsing responses according to schema is better. Sending requests to the server is orthogonal to the purpose.&lt;p&gt;What we&amp;#x27;ve found useful in practice in dealing with similar problems:&lt;p&gt;- Use json5 instead of json when parsing. It allows trailing commas.&lt;p&gt;- Don&amp;#x27;t let it respond in true&amp;#x2F;false. Instead, ask it for a short sentence explaining whether it is true or false. Afterwards, use a small embedding model such as sbert to extract true&amp;#x2F;false from the sentence. We&amp;#x27;ve found that GPT is able to reason better in this case, and it is much more robust.&lt;p&gt;- For numerical scores, do a similar thing by asking GPT for a description, then with the small embedding model write a few examples matching your score scale, and for each response use the score of the best matched example. If you let GPT give you scores directly without explanation, 20% of the time it will give you nonsense.</text></item></parent_chain><comment><author>icyfox</author><text>Thanks for the thoughts! I&amp;#x27;ve deployed a few meta models that act like you&amp;#x27;re describing for second stage predictions, but for fuzzy task definitions have actually seen similar luck with having GPT explicitly explain its rational and then force it to choose a true&amp;#x2F;false rating. My payloads often end up looking like:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  class Payload:
    reasoning: str = Field(description=&amp;quot;Why this value might be true or false),
    answer: bool
&lt;/code&gt;&lt;/pre&gt;
Since it&amp;#x27;s autoregressive I imagine the schema helps to define the universe of what it&amp;#x27;s supposed to do, then the decoder attention when it&amp;#x27;s filling the `answer` can look back on the reasoning and weigh the sentiment internally. I imagine the accuracy specifics depend a lot on the end deployment here.</text></comment>