<story><title>Table Detection and Extraction Using Deep Learning</title><url>https://nanonets.com/blog/table-extraction-deep-learning/</url></story><parent_chain><item><author>willvarfar</author><text>I&amp;#x27;ve worked with several companies that try to parse things in PDF documents, extracting tables and paragraphs etc.  This is actually challenging because a PDF is a large bag of words and fragments of words with x y positions.  There is a particularly popular word processor that emits individual characters.  Just determining that two fragments are part of the same word is challenging as is detecting bullet points, etc.&lt;p&gt;The AI approaches are definitely still worse than human-written rules.  I can infer - and I&amp;#x27;ve chatted with the devs to confirm - from the quality of the text and table extraction whether the company is using a modern NN approach or someone has sat down and handwritten some simple rules that understand indents and baselines etc.</text></item></parent_chain><comment><author>tlarkworthy</author><text>Yes exactly, table comprehension is a logic driven, non-local inference problem. Critically, its the non-locality that trips up common machine learning techniques. I wrote an approach using mixed integer programming once[1]&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;edinburghhacklab.com&amp;#x2F;2013&amp;#x2F;09&amp;#x2F;probabalistic-scraping-of-plain-text-tables&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;edinburghhacklab.com&amp;#x2F;2013&amp;#x2F;09&amp;#x2F;probabalistic-scraping-...&lt;/a&gt;</text></comment>
<story><title>Table Detection and Extraction Using Deep Learning</title><url>https://nanonets.com/blog/table-extraction-deep-learning/</url></story><parent_chain><item><author>willvarfar</author><text>I&amp;#x27;ve worked with several companies that try to parse things in PDF documents, extracting tables and paragraphs etc.  This is actually challenging because a PDF is a large bag of words and fragments of words with x y positions.  There is a particularly popular word processor that emits individual characters.  Just determining that two fragments are part of the same word is challenging as is detecting bullet points, etc.&lt;p&gt;The AI approaches are definitely still worse than human-written rules.  I can infer - and I&amp;#x27;ve chatted with the devs to confirm - from the quality of the text and table extraction whether the company is using a modern NN approach or someone has sat down and handwritten some simple rules that understand indents and baselines etc.</text></item></parent_chain><comment><author>crispyambulance</author><text>It was really shocking when I learned that the way pdf works is as you describe, literally fragments of text with positions and essentially no semantics.&lt;p&gt;I think a lot of folks find this out as I did, when they run into a project where they need to extract info from pdf documents. Without knowing anything about pdf, one can easily assume that it will be possible to do things like &amp;quot;can&amp;#x27;t we just extract some semantic structures like headings, tables, etc&amp;quot;... but nooo, it don&amp;#x27;t work that way!&lt;p&gt;Discovering the true nature of pdf is major WTF moment because we&amp;#x27;re so conditioned to expect documents to have a semantic structure. It&amp;#x27;s hard to understand how a standard can take the exact opposite approach and be so successful.</text></comment>