<story><title>Celery – Best Practices</title><url>https://denibertovic.com/posts/celery-best-practices/</url></story><parent_chain></parent_chain><comment><author>xenator</author><text>In many projects Celery is overkill. Common scenario I saw:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  1. We have problem, lets use Celery
  2. Now we have one more problem.
&lt;/code&gt;&lt;/pre&gt;
I found &lt;a href=&quot;http://python-rq.org/&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;python-rq.org&amp;#x2F;&lt;/a&gt; much more handy and cover most cases. It uses redis as query broker. Flask, Django integration included &lt;a href=&quot;https://github.com/mattupstate/flask-rq/&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;mattupstate&amp;#x2F;flask-rq&amp;#x2F;&lt;/a&gt; &lt;a href=&quot;https://github.com/ui/django-rq&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;ui&amp;#x2F;django-rq&lt;/a&gt;</text></comment>
<story><title>Celery – Best Practices</title><url>https://denibertovic.com/posts/celery-best-practices/</url></story><parent_chain></parent_chain><comment><author>mickeyp</author><text>Good, basic practices to follow. Here&amp;#x27;s a few more:&lt;p&gt;- If you&amp;#x27;re using AMQP&amp;#x2F;RabbitMQ as your result back end it will create a lot of dead queues to store results in. This can easily overwhelm your RabbitMQ server if you don&amp;#x27;t clear these out frequently. Newer releases of Celery will do this daily I think - but it&amp;#x27;s worth keeping in mind if your RMQ instance falls over in prod.&lt;p&gt;- Use chaining to build up &amp;quot;sequential&amp;quot; tasks that need doing instead of calling one after another in the same task (or worse, doing a big mouthful of work) in one task as Celery can prioritise many tasks better than synchronously calling several tasks in a row from one &amp;quot;master&amp;quot; task.&lt;p&gt;- Try to keep a consistent module import pattern for celery tasks, or explicitly name them, as Celery does a lot of magic in the background so task spawning is seamless to the developer. This is very important as you should never mix relative and absolute importing when you are dealing with tasks. from foo import mytask may be picked up differently than &amp;quot;import foo&amp;quot; followed by &amp;quot;foo.mytask&amp;quot; would resulting in some tasks not being picked up by Celery(!)&lt;p&gt;- Never pass database objects, as OP says, is true; but go one step further and don&amp;#x27;t pass complex objects at all if you can avoid it. I vaguely remember some of the urllib&amp;#x2F;httplib exceptions in Python not being serializable and causing very cryptic errors if you didn&amp;#x27;t capture the exception and sanitise it or re-raise your own.&lt;p&gt;- Use proper configuration management to set up and configure Celery plus what ever messaging broker&amp;#x2F;backend. There&amp;#x27;s nothing more frustrating than spending your time trying to replicate somebody&amp;#x27;s half-assed Celery&amp;#x2F;Rabbit configuration that they didn&amp;#x27;t nail down and test properly in a clean-room environment.</text></comment>