<story><title>Ask HN: Can web scraping be the basis of a viable business model?</title><text>I&amp;#x27;m a data engineer at heart, and I never did or enjoyed front-end work. Having said that I always was happy to code and evolve crawlers and web scrapers. Now I&amp;#x27;ve taken some time off from work and gigs and I&amp;#x27;m working on a  side-project  I&amp;#x27;ve been hacking for some time.&lt;p&gt;Without getting into the details yet: it aims to make web data collection a little bit easier for non-devs. I&amp;#x27;ll soon have an MVP and will start pitching to investors: aiming for an open-source business model (after a few months of stealth development) and eventually a typical SaaS offering for extra functionality.&lt;p&gt;At this point I&amp;#x27;m trying to consolidate and counter the steel-man counter-arguments I should expect from investors.  The most obvious one: as one can imagine, the product it&amp;#x27;s not magic and, after a certain point it does require some manual work from the customer, hence this is an aspect I should prepare for.&lt;p&gt;I have done some preliminary analysis of the space of potential competitors (think import.io, Apify, Zyte&amp;#x2F;ScarpingHub, etc.) and described opportunities for differentiation. What I&amp;#x27;m afraid of is getting sidetracked in a discussion of &amp;quot;um, this is web scraping and it&amp;#x27;s hard to make a business on top of it&amp;quot;.&lt;p&gt;I understand that there&amp;#x27;s not much context now and one could easily say &amp;quot;well yeah, anything could be possible with a good team, product...&amp;quot;, but I&amp;#x27;m reaching out to the HN community to gather some considerations, mental models and pointers, I may not think of myself at this point.</text></story><parent_chain><item><author>apienx</author><text>Google, a trillion dollar company, is essentially the world&amp;#x27;s largest web scraper. So...yes! You&amp;#x27;ll almost certainly find a way to monetize that.&lt;p&gt;Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There&amp;#x27;s tremendous value in restoring some of it.</text></item></parent_chain><comment><author>asdfasgasdgasdg</author><text>&amp;gt; Google, a trillion dollar company, is essentially the world&amp;#x27;s largest web scraper.&lt;p&gt;Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It&amp;#x27;s beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying that a person is, in essence, the world&amp;#x27;s largest mouth.&lt;p&gt;I think it&amp;#x27;s incredibly difficult to build a profitable business in this space. The number of customers who a) need to scrape the web b) aren&amp;#x27;t sophisticated enough to do it themselves and c) are big enough to make $$$ from are small. The important bit is always processing the web pages for whatever content is salient to the given customer. Which means that you need to deliver the web pages to them. So effectively your business is providing nothing more than URL lists and potentially some additional metadata compared to what the customer would get if they fetched the pages themselves. There are definitely some other complexities you could resolve, but it&amp;#x27;s hard to imagine that those benefits would be enough to build a business on.</text></comment>
<story><title>Ask HN: Can web scraping be the basis of a viable business model?</title><text>I&amp;#x27;m a data engineer at heart, and I never did or enjoyed front-end work. Having said that I always was happy to code and evolve crawlers and web scrapers. Now I&amp;#x27;ve taken some time off from work and gigs and I&amp;#x27;m working on a  side-project  I&amp;#x27;ve been hacking for some time.&lt;p&gt;Without getting into the details yet: it aims to make web data collection a little bit easier for non-devs. I&amp;#x27;ll soon have an MVP and will start pitching to investors: aiming for an open-source business model (after a few months of stealth development) and eventually a typical SaaS offering for extra functionality.&lt;p&gt;At this point I&amp;#x27;m trying to consolidate and counter the steel-man counter-arguments I should expect from investors.  The most obvious one: as one can imagine, the product it&amp;#x27;s not magic and, after a certain point it does require some manual work from the customer, hence this is an aspect I should prepare for.&lt;p&gt;I have done some preliminary analysis of the space of potential competitors (think import.io, Apify, Zyte&amp;#x2F;ScarpingHub, etc.) and described opportunities for differentiation. What I&amp;#x27;m afraid of is getting sidetracked in a discussion of &amp;quot;um, this is web scraping and it&amp;#x27;s hard to make a business on top of it&amp;quot;.&lt;p&gt;I understand that there&amp;#x27;s not much context now and one could easily say &amp;quot;well yeah, anything could be possible with a good team, product...&amp;quot;, but I&amp;#x27;m reaching out to the HN community to gather some considerations, mental models and pointers, I may not think of myself at this point.</text></story><parent_chain><item><author>apienx</author><text>Google, a trillion dollar company, is essentially the world&amp;#x27;s largest web scraper. So...yes! You&amp;#x27;ll almost certainly find a way to monetize that.&lt;p&gt;Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There&amp;#x27;s tremendous value in restoring some of it.</text></item></parent_chain><comment><author>elliottinvent</author><text>&amp;gt; Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable.&lt;p&gt;Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag.&lt;p&gt;&amp;gt; There&amp;#x27;s tremendous value in restoring some of it.&lt;p&gt;To this comment and OP, my startup is using web scraping to pre-populate machine-readable data for a DNS-based protocol called NUM [0]. So as others have said, whilst web scraping itself may be difficult to build into a viable business, it can be a key component of a viable business. Email in my profile if you want to discuss.&lt;p&gt;0. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;num.uk&amp;#x2F;blog&amp;#x2F;we-crawled-5m-uk-websites-and-published-23m-data-points-to-dns&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;num.uk&amp;#x2F;blog&amp;#x2F;we-crawled-5m-uk-websites-and-published-...&lt;/a&gt;</text></comment>