<story><title>Show HN: Crawlee for Python – a web scraping and browser automation library</title><url>https://crawlee.dev/python/</url><text>Hey all,&lt;p&gt;This is Jan, the founder of Apify (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;apify.com&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;apify.com&amp;#x2F;&lt;/a&gt;) — a full-stack web scraping platform. After the success of Crawlee for JavaScript (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apify&amp;#x2F;crawlee&amp;#x2F;&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apify&amp;#x2F;crawlee&amp;#x2F;&lt;/a&gt;) and the demand from the Python community, we&amp;#x27;re launching Crawlee for Python today!&lt;p&gt;The main features are:&lt;p&gt;- A unified programming interface for both HTTP (HTTPX with BeautifulSoup) &amp;amp; headless browser crawling (Playwright)&lt;p&gt;- Automatic parallel crawling based on available system resources&lt;p&gt;- Written in Python with type hints for enhanced developer experience&lt;p&gt;- Automatic retries on errors or when you’re getting blocked&lt;p&gt;- Integrated proxy rotation and session management&lt;p&gt;- Configurable request routing - direct URLs to the appropriate handlers&lt;p&gt;- Persistent queue for URLs to crawl&lt;p&gt;- Pluggable storage for both tabular data and files&lt;p&gt;For details, you can read the announcement blog post: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;crawlee.dev&amp;#x2F;blog&amp;#x2F;launching-crawlee-python&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;crawlee.dev&amp;#x2F;blog&amp;#x2F;launching-crawlee-python&lt;/a&gt;&lt;p&gt;Our team and I will be happy to answer here any questions you might have.</text></story><parent_chain></parent_chain><comment><author>mdaniel</author><text>You&amp;#x27;ll want to prioritize documenting the &lt;i&gt;existing&lt;/i&gt; features, since it&amp;#x27;s no good having a super awesome full stack web scraping platform if only you can use it. I ordinarily would default to a &amp;quot;read the source&amp;quot; response but your cutesy coding style makes that a non-starter&lt;p&gt;As a concrete example: command-f for &amp;quot;tier&amp;quot; on &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;crawlee.dev&amp;#x2F;python&amp;#x2F;docs&amp;#x2F;guides&amp;#x2F;proxy-management&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;crawlee.dev&amp;#x2F;python&amp;#x2F;docs&amp;#x2F;guides&amp;#x2F;proxy-management&lt;/a&gt; and tell me how anyone could possibly know what `tiered_proxy_urls: list[list[str]] | None = None` should contain and why?</text></comment>
<story><title>Show HN: Crawlee for Python – a web scraping and browser automation library</title><url>https://crawlee.dev/python/</url><text>Hey all,&lt;p&gt;This is Jan, the founder of Apify (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;apify.com&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;apify.com&amp;#x2F;&lt;/a&gt;) — a full-stack web scraping platform. After the success of Crawlee for JavaScript (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apify&amp;#x2F;crawlee&amp;#x2F;&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apify&amp;#x2F;crawlee&amp;#x2F;&lt;/a&gt;) and the demand from the Python community, we&amp;#x27;re launching Crawlee for Python today!&lt;p&gt;The main features are:&lt;p&gt;- A unified programming interface for both HTTP (HTTPX with BeautifulSoup) &amp;amp; headless browser crawling (Playwright)&lt;p&gt;- Automatic parallel crawling based on available system resources&lt;p&gt;- Written in Python with type hints for enhanced developer experience&lt;p&gt;- Automatic retries on errors or when you’re getting blocked&lt;p&gt;- Integrated proxy rotation and session management&lt;p&gt;- Configurable request routing - direct URLs to the appropriate handlers&lt;p&gt;- Persistent queue for URLs to crawl&lt;p&gt;- Pluggable storage for both tabular data and files&lt;p&gt;For details, you can read the announcement blog post: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;crawlee.dev&amp;#x2F;blog&amp;#x2F;launching-crawlee-python&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;crawlee.dev&amp;#x2F;blog&amp;#x2F;launching-crawlee-python&lt;/a&gt;&lt;p&gt;Our team and I will be happy to answer here any questions you might have.</text></story><parent_chain></parent_chain><comment><author>Findecanor</author><text>Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags?
These are getting more important now, especially in the EU after the DSM directive.</text></comment>