<story><title>Faster neural networks straight from JPEG (2018)</title><url>https://www.uber.com/blog/neural-networks-jpeg/</url></story><parent_chain><item><author>mochomocha</author><text>I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they&amp;#x27;re ~all zeros) and using a neural network library with fast ops on sparse data.&lt;p&gt;Training on pixels is inefficient. Why have your first layers of CNNs relearn what&amp;#x27;s already smartly encoded in the JPEG bits in the first place before it&amp;#x27;s blown into a bloated  height x width x 3 float matrix?</text></item></parent_chain><comment><author>corysama</author><text>As an AI armchair quarterback, I&amp;#x27;ve always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. &amp;quot;Look what this method can learn from raw data completely independently!&amp;quot; makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone learns from the papers, so that&amp;#x27;s assumed to be the standard practice.</text></comment>
<story><title>Faster neural networks straight from JPEG (2018)</title><url>https://www.uber.com/blog/neural-networks-jpeg/</url></story><parent_chain><item><author>mochomocha</author><text>I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they&amp;#x27;re ~all zeros) and using a neural network library with fast ops on sparse data.&lt;p&gt;Training on pixels is inefficient. Why have your first layers of CNNs relearn what&amp;#x27;s already smartly encoded in the JPEG bits in the first place before it&amp;#x27;s blown into a bloated  height x width x 3 float matrix?</text></item></parent_chain><comment><author>johntb86</author><text>I would worry that the fixed, non-overlapping block nature of a JPEG would reduce translation invariance - shift an image by 4 pixels and the DCT coefficients may look very different. People have been doing a lot of work to try to reduce the dependence of the image on the actual pixel coordinates - see for example &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;research.nvidia.com&amp;#x2F;publication&amp;#x2F;2021-12_alias-free-generative-adversarial-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;research.nvidia.com&amp;#x2F;publication&amp;#x2F;2021-12_alias-free-g...&lt;/a&gt;</text></comment>