A failsafe trigger was enabled when our background instances ran out of filesystem space. We expanded the filesystem and turned the failsafe off while we catch up with background jobs.
We have since turned the failsafe back on and everything should be back to normal.
We are running a post mortem in the morning to make sure we have better safeguard so we can catch this kind of issues before it happens next time.
We have found the problem and applied a fix. We are monitoring our background jobs.
Subscribe to updates for Investigating issue regarding processing of background jobs via email and/or text message. Whenever an update is posted, you'll receive a notification.