<story><title>Pix2tex: Using a ViT to convert images of equations into LaTeX code</title><url>https://github.com/lukas-blecher/LaTeX-OCR</url></story><parent_chain></parent_chain><comment><author>kuter</author><text>Took a peek at the models they use. It seems to be a vision transformer encoder decoder architecture with a resent backbone. Looks really good. I had a similar idea of training a model and making a desktop application, but haven&amp;#x27;t had the opportunity. I wonder how much compute it took to train the model.&lt;p&gt;I think this paper was the first one to do OCR on LaTeX: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;cs231n.stanford.edu&amp;#x2F;reports&amp;#x2F;2017&amp;#x2F;pdfs&amp;#x2F;815.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http:&amp;#x2F;&amp;#x2F;cs231n.stanford.edu&amp;#x2F;reports&amp;#x2F;2017&amp;#x2F;pdfs&amp;#x2F;815.pdf&lt;/a&gt;
The paper describes an Encoder-Decoder architecture with CNN encoder and LSTM based decoder.</text></comment>
<story><title>Pix2tex: Using a ViT to convert images of equations into LaTeX code</title><url>https://github.com/lukas-blecher/LaTeX-OCR</url></story><parent_chain></parent_chain><comment><author>perihelions</author><text>For the morbidly curious, that nightmare math is someone&amp;#x27;s quantum field theory notes which they typeset in TeX:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;rohankulkarni.me&amp;#x2F;files&amp;#x2F;notes&amp;#x2F;heidelberg_qft&amp;#x2F;12_2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;rohankulkarni.me&amp;#x2F;files&amp;#x2F;notes&amp;#x2F;heidelberg_qft&amp;#x2F;12_2.pdf&lt;/a&gt; (&lt;i&gt;&amp;quot;12.2 Diagrammatic expansion of partition function for Yukawa theory&amp;quot;&lt;/i&gt;)</text></comment>