<story><title>What&apos;s so hard about PDF text extraction?</title><url>https://www.filingdb.com/pdf-text-extraction</url></story><parent_chain><item><author>tolmasky</author><text>What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.</text></item><item><author>Alex3917</author><text>&amp;gt; I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a &amp;quot;block&amp;quot; (paragraph), and I got a huge explanation on how hard it would be.&lt;p&gt;The funny thing is that creating a universal algorithm to convert PDFs and&amp;#x2F;or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as much profit to any company that can solve it. But there are hundreds of billions of dollars going into self-driving cars, and like zero dollars going into this problem.</text></item><item><author>tolmasky</author><text>This is why iPhone didn&amp;#x27;t initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a &amp;quot;block&amp;quot; (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted.&lt;p&gt;Edit: To add a little more color, given that none of us was (or at least certainly I wasn&amp;#x27;t) an expert on the PDF format, we had so far treated the bug like a bug of probably at-most moderate complexity (just have to read up on PDF and figure out what the base unit is or whatever). After discovering what this article talks about, it became evident that any solution we cobbled together in the time we had left would really just be signing up for an endless stream of it-doesn&amp;#x27;t-work-quite-right bugs. So, a feature that would become a bug emitter. I remember in particular considering one of the main use cases: scientific articles that are usually in two columns, AND also used justified text. A lot of times the spaces between words could be as large as the spaces between columns, so the statistical &amp;quot;grouping&amp;quot; of characters to try to identify the &amp;quot;macro rectangle&amp;quot; shape could get tricky without severely special-casing for this. All this being said, as the story should make clear, I put about one day of thought into this before the decision was made to avoid it for 1.0, so far all I know there are actually really good solutions to this. Even writing this now I am starting to think of fun ways to deal with this, but at the time, it was one of a huge list of things that needed to get done and had been underestimated in complexity.</text></item></parent_chain><comment><author>greycol</author><text>Almost any business that has physical suppliers or business customers.&lt;p&gt;PDF is de-facto standard for any invoicing, POs, quotes, etc.&lt;p&gt;If you solve the problem you can effectively programmatically deal with invoicing&amp;#x2F;payments&amp;#x2F; large parts of ordering&amp;#x2F;dispensing. It&amp;#x27;s a no brainer to add it on to almost any financial&amp;#x2F;procurement software that deals with inter business stuff.&lt;p&gt;Any small-medium physical business can probably half their financial department if you can dependably solve this issue.</text></comment>
<story><title>What&apos;s so hard about PDF text extraction?</title><url>https://www.filingdb.com/pdf-text-extraction</url></story><parent_chain><item><author>tolmasky</author><text>What are the groups that would benefit most from the PDF-to-HTML conversion? Who are the customers that would drive this profit? I tried to make those sentences not sound contentious but unfortunately they do, but I am genuinely curious about this space and who is feeling the lack of this technology most.</text></item><item><author>Alex3917</author><text>&amp;gt; I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a &amp;quot;block&amp;quot; (paragraph), and I got a huge explanation on how hard it would be.&lt;p&gt;The funny thing is that creating a universal algorithm to convert PDFs and&amp;#x2F;or HTML to plaintext is probably comparable in difficulty to building level 5 self-driving cars, and would accrue at least as much profit to any company that can solve it. But there are hundreds of billions of dollars going into self-driving cars, and like zero dollars going into this problem.</text></item><item><author>tolmasky</author><text>This is why iPhone didn&amp;#x27;t initially ship with double-tap to zoom for PDF paragraphs (like it had for blocks on web pages). I know because I was assigned the feature, and I went over to the PDF guy to ask how I would determine on an arbitrary PDF what was probably a &amp;quot;block&amp;quot; (paragraph), and I got a huge explanation on how hard it would be. I relayed this to my manager and the bug was punted.&lt;p&gt;Edit: To add a little more color, given that none of us was (or at least certainly I wasn&amp;#x27;t) an expert on the PDF format, we had so far treated the bug like a bug of probably at-most moderate complexity (just have to read up on PDF and figure out what the base unit is or whatever). After discovering what this article talks about, it became evident that any solution we cobbled together in the time we had left would really just be signing up for an endless stream of it-doesn&amp;#x27;t-work-quite-right bugs. So, a feature that would become a bug emitter. I remember in particular considering one of the main use cases: scientific articles that are usually in two columns, AND also used justified text. A lot of times the spaces between words could be as large as the spaces between columns, so the statistical &amp;quot;grouping&amp;quot; of characters to try to identify the &amp;quot;macro rectangle&amp;quot; shape could get tricky without severely special-casing for this. All this being said, as the story should make clear, I put about one day of thought into this before the decision was made to avoid it for 1.0, so far all I know there are actually really good solutions to this. Even writing this now I am starting to think of fun ways to deal with this, but at the time, it was one of a huge list of things that needed to get done and had been underestimated in complexity.</text></item></parent_chain><comment><author>Alex3917</author><text>So I have a lot of experience with basically the same problem just from working on this: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.prettyfwd.com&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.prettyfwd.com&lt;/a&gt;. As an example of the opportunity size just in the email domain, the amount of personal non-spam email sent every day is like 100x the total size of Wikipedia, but nothing is really done with any of this information because of this challenge. Basically applications are things like:&lt;p&gt;- Better search engine results&lt;p&gt;- Identifying experts within a company&lt;p&gt;- Better machine translation&lt;p&gt;- Finding accounting fraud&lt;p&gt;- Automating legal processes&lt;p&gt;For context, the reason why Facebook is the most successful social network is that they&amp;#x27;re able to turn behavioral residue into content. If you can get better at taking garbage data and repackaging it into something useful, it stands to reason that there are lots of other companies the size of Facebook that can be created.</text></comment>