<story><title>Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version</title><text>Hey HN - this is Alexander and Gabe from Hatchet (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;hatchet.run&quot;&gt;https:&amp;#x2F;&amp;#x2F;hatchet.run&lt;/a&gt;). We’re building a modern task queue as an alternative to tools like Celery for Python and BullMQ for Node. Our open-source repo is at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hatchet-dev&amp;#x2F;hatchet&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hatchet-dev&amp;#x2F;hatchet&lt;/a&gt; and is 100% MIT licensed.&lt;p&gt;When we did a Show HN a few months ago (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=39643136&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=39643136&lt;/a&gt;), our cloud version was invite-only and we were focused on our open-source offering.&lt;p&gt;Today we’re launching our self-serve cloud so that anyone can get started creating tasks on our platform - you can get started at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;cloud.onhatchet.run&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;cloud.onhatchet.run&lt;/a&gt;, or you can use these credentials to access a demo (should be prefilled):&lt;p&gt;&lt;pre&gt;&lt;code&gt;  URL: https:&amp;#x2F;&amp;#x2F;demo.hatchet-tools.com 
  Email: hacker@news.ycombinator.com
  Password: HatchetDemo123!
&lt;/code&gt;&lt;/pre&gt;
People are currently using Hatchet for a bunch of use-cases: orchestrating RAG pipelines, queueing up user notifications, building agentic LLM workflows, or scheduling image generation tasks on GPUs.&lt;p&gt;We built this out of frustration with existing tools and a conviction that PostgreSQL is the right choice for a task queue. Beyond the fact that many developers are already using Postgres in their stack, which makes it easier to self-host Hatchet, it’s also easier to model higher-order concepts in Postgres, like chains of tasks (which we call workflows). In our system, the acknowledgement of the task, the task result, and the updates to higher-order models are done as part of the same Postgres transaction, which significantly reduces the risk of data loss&amp;#x2F;race conditions when compared with other task queues (which usually pass acknowledgements through a broker, storing the task results elsewhere, and only then figuring out the next task in the chain).&lt;p&gt;We also became increasingly frustrated with tools like Celery and the challenges it introduces when using a modern Python stack (&amp;gt; 3.5). We wrote up a list of these frustrations here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;blog&amp;#x2F;problems-with-celery&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;blog&amp;#x2F;problems-with-celery&lt;/a&gt;.&lt;p&gt;Since our Show HN, we’ve (partially or completely) addressed the most common pieces of feedback from the post, which we’ll outline here:&lt;p&gt;1. The most common ask was built-in support for fanout workflows — one task which triggers an arbitrary number of child tasks to run in parallel. We previously only had support for DAG executions. We generalized this concept and launched child workflows (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;child-workflows&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;child-workflows&lt;/a&gt;). This is the first step towards a developer-friendly model of durable execution.&lt;p&gt;2. Support for HTTP-based triggers — we’ve built out support for webhook workers (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;webhooks&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;webhooks&lt;/a&gt;), which allow you to trigger any workflow over an HTTP webhook. This is particularly useful for apps on Vercel, who are dealing with timeout limits of 60s, 300s, or 900s (depending on your tier).&lt;p&gt;3. Our RabbitMQ dependency — while we haven’t gotten rid of this completely, we’ve recently launched hatchet-lite (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;self-hosting&amp;#x2F;hatchet-lite&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;self-hosting&amp;#x2F;hatchet-lite&lt;/a&gt;), which allows you to run the various Hatchet components in a single Docker image that bundles RabbitMQ along with a migration process, admin CLI, our REST API, and our gRPC engine. Hopefully the lite was a giveaway, but this is meant for local development and low-volume processing, on the order of hundreds per minute.&lt;p&gt;We’ve also launched more features, like support for global rate limiting, steps which only run on workflow failure, and custom event streaming.&lt;p&gt;We’ll be here the whole day for questions and feedback, and look forward to hearing your thoughts!</text></story><parent_chain><item><author>krick</author><text>Can somebody explain why would I use it instead a simple Redis&amp;#x2F;SQS&amp;#x2F;Postgres queue implemented in 50 LOC (+ some grafana panel for monitoring) (which is pretty much mandatory even for a wrapper of this or any other service)? I&amp;#x27;m not trying to mock it, it&amp;#x27;s a serious question. What is implied by &amp;quot;task queue&amp;quot; that makes it worth bothering to use a dedicated service?</text></item></parent_chain><comment><author>abelanger</author><text>You&amp;#x27;re right, if all you need is a queue with a small number of workers connected at low volume, you don&amp;#x27;t need Hatchet or any other managed queue - you can get some pretty performant behavior with something like: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;abelanger5&amp;#x2F;postgres-fair-queue&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;internal&amp;#x2F;dbsqlc&amp;#x2F;tasks.sql&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;abelanger5&amp;#x2F;postgres-fair-queue&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;...&lt;/a&gt;.&lt;p&gt;The point of Hatchet is to support more complex behavior - like chaining tasks together, building automation around querying and retrying failed tasks, handling a lot of the fairness and concurrency use-cases you&amp;#x27;d otherwise need to build yourself, etc - or just getting something that works out of the box and can support those use-cases in the future.&lt;p&gt;And if you are running at low volume and trying to debug user issues, a grafana panel isn&amp;#x27;t going to get you the level of granularity or admin control you need to track down the errors in your methods (rather than just at the queue level). You&amp;#x27;d need to integrate your task queue with Sentry and a logging system - and in our case, error tracing and logging are available in the Hatchet UI.</text></comment>
<story><title>Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version</title><text>Hey HN - this is Alexander and Gabe from Hatchet (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;hatchet.run&quot;&gt;https:&amp;#x2F;&amp;#x2F;hatchet.run&lt;/a&gt;). We’re building a modern task queue as an alternative to tools like Celery for Python and BullMQ for Node. Our open-source repo is at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hatchet-dev&amp;#x2F;hatchet&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hatchet-dev&amp;#x2F;hatchet&lt;/a&gt; and is 100% MIT licensed.&lt;p&gt;When we did a Show HN a few months ago (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=39643136&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=39643136&lt;/a&gt;), our cloud version was invite-only and we were focused on our open-source offering.&lt;p&gt;Today we’re launching our self-serve cloud so that anyone can get started creating tasks on our platform - you can get started at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;cloud.onhatchet.run&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;cloud.onhatchet.run&lt;/a&gt;, or you can use these credentials to access a demo (should be prefilled):&lt;p&gt;&lt;pre&gt;&lt;code&gt;  URL: https:&amp;#x2F;&amp;#x2F;demo.hatchet-tools.com 
  Email: hacker@news.ycombinator.com
  Password: HatchetDemo123!
&lt;/code&gt;&lt;/pre&gt;
People are currently using Hatchet for a bunch of use-cases: orchestrating RAG pipelines, queueing up user notifications, building agentic LLM workflows, or scheduling image generation tasks on GPUs.&lt;p&gt;We built this out of frustration with existing tools and a conviction that PostgreSQL is the right choice for a task queue. Beyond the fact that many developers are already using Postgres in their stack, which makes it easier to self-host Hatchet, it’s also easier to model higher-order concepts in Postgres, like chains of tasks (which we call workflows). In our system, the acknowledgement of the task, the task result, and the updates to higher-order models are done as part of the same Postgres transaction, which significantly reduces the risk of data loss&amp;#x2F;race conditions when compared with other task queues (which usually pass acknowledgements through a broker, storing the task results elsewhere, and only then figuring out the next task in the chain).&lt;p&gt;We also became increasingly frustrated with tools like Celery and the challenges it introduces when using a modern Python stack (&amp;gt; 3.5). We wrote up a list of these frustrations here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;blog&amp;#x2F;problems-with-celery&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;blog&amp;#x2F;problems-with-celery&lt;/a&gt;.&lt;p&gt;Since our Show HN, we’ve (partially or completely) addressed the most common pieces of feedback from the post, which we’ll outline here:&lt;p&gt;1. The most common ask was built-in support for fanout workflows — one task which triggers an arbitrary number of child tasks to run in parallel. We previously only had support for DAG executions. We generalized this concept and launched child workflows (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;child-workflows&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;child-workflows&lt;/a&gt;). This is the first step towards a developer-friendly model of durable execution.&lt;p&gt;2. Support for HTTP-based triggers — we’ve built out support for webhook workers (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;webhooks&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;home&amp;#x2F;features&amp;#x2F;webhooks&lt;/a&gt;), which allow you to trigger any workflow over an HTTP webhook. This is particularly useful for apps on Vercel, who are dealing with timeout limits of 60s, 300s, or 900s (depending on your tier).&lt;p&gt;3. Our RabbitMQ dependency — while we haven’t gotten rid of this completely, we’ve recently launched hatchet-lite (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;self-hosting&amp;#x2F;hatchet-lite&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.hatchet.run&amp;#x2F;self-hosting&amp;#x2F;hatchet-lite&lt;/a&gt;), which allows you to run the various Hatchet components in a single Docker image that bundles RabbitMQ along with a migration process, admin CLI, our REST API, and our gRPC engine. Hopefully the lite was a giveaway, but this is meant for local development and low-volume processing, on the order of hundreds per minute.&lt;p&gt;We’ve also launched more features, like support for global rate limiting, steps which only run on workflow failure, and custom event streaming.&lt;p&gt;We’ll be here the whole day for questions and feedback, and look forward to hearing your thoughts!</text></story><parent_chain><item><author>krick</author><text>Can somebody explain why would I use it instead a simple Redis&amp;#x2F;SQS&amp;#x2F;Postgres queue implemented in 50 LOC (+ some grafana panel for monitoring) (which is pretty much mandatory even for a wrapper of this or any other service)? I&amp;#x27;m not trying to mock it, it&amp;#x27;s a serious question. What is implied by &amp;quot;task queue&amp;quot; that makes it worth bothering to use a dedicated service?</text></item></parent_chain><comment><author>vasco</author><text>You can use celery with postgres without issues if you want the stuff you don&amp;#x27;t get with that, like tweakable retries, tweakable amounts of prefetch and other important-at-scale things. Plus out of the box working sdk with higher level patterns for you developers. Like what if devs want to track how long something waited in the queue or a metric about retries etc, things that you&amp;#x27;d have to roll by hand.</text></comment>