How to perform healthcheck of Celery beat?
Unpredictability or manual validation is the worst enemy of a developer.
When we deploy any application to production our first hunch is to check if everything is working as expected. We start adding automatic health-check for all the systems in place so that we don't have to manually verify every system daily and get notified whenever we have any system issues.
In my previous blog, I mentioned how you can integrate health-check for any Django and Celery applications. Go through the blog, if you have not integrated a health check for your Django and celery web application already.
The above blog shows you to monitor Django and Celery tasks, but what if your celery beat is not functioning as per the expected way. Since celery beat is used to create periodic tasks, so it is highly unlikely that you would know about the failure of the celery beat system, from someone complaining about an anomaly.
Lets us now solve see how we can integrate health checks for periodic tasks added inside celery beat. One of the solutions which I prefer is, perform push health-check or reverse monitoring. Statuscake provides push health check out of the box.
Push tests or reverse monitoring is the process when our server pings the status cake server periodically, and if StatusCake doesn't receive a particular ping during the given interval, then it would raise an alert.
What we should do?
We should create a new celery periodic task and ping the StatusCake every 5 min to notify that, celery beat is working as expected.
@app.on_after_finalize.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(crontab(minute="*/10"), ping_status_cake.s())
@task
def ping_status_cake():
# Ping status cakes push link every 10 min to check celery beat,
# we will get notified from statuscake on any failure.
STATUS_PUSH_LINK = 'https://push.statuscake.com/?PK=abc&TestID=0'
requests.get(STATUS_PUSH_LINK)
Add the above code, in tasks.py, also please extract STATUS_PUSH_LINK from the StatusCake dashboard.
If you want to know how you can create periodic tasks using celery Beat, please follow my other blog on this topic.
