Monitoring and Reporting Errors from Sidekiq Jobs

Sidekiq is famous (in the Ruby world) as an easy and simple background job processing solution. Sidekiq covers almost all the use cases that you expect from a background job processing solution but sometimes you need something that is not supported out of the box. That’s when Sidekiq middleware comes in the picture, middleware can extend Sidekiq to run your code before and during job life cycle.

We, at Clarisights, use Sidekiq extensively for background jobs. Just like any other piece of software, sometimes background jobs fail. When that happens we want to know why and how that happened.

Why do we need this?

We wanted to capture every unhandled error from background jobs. We collect all errors in our central error reporting tool (Sentry). Our application is configured to use sentry but we were only getting errors when all retries for a job were exhausted. It was happening because Sidekiq captures any exception occurred in a job (that’s how it decides that a job failed) and uses it to retry failed jobs. It doesn’t raise after capture unless retries are exhausted which will happen after a lot of retries (you can configure retry policy).

Sidekiq Sentry Middleware

Sidekiq has notion of server-side and client-side middleware, Server-side middleware runs around job life cycle while client-side middleware runs before pushing a job to Redis.

We wrote this server-side middleware to report unhandled errors from jobs. If a job fails, our middleware will capture an unhandled exception and report that to Sentry (can be easily adapted to report to other sources as well). After reporting, we will raise that exception so Sidekiq & other middlewares down the chain know that the job failed.

Raising captured exception again is important here because, if you don’t raise, Sidekiq or any other middleware down the chain will think that job completed successfully.

# Sidekiq Middleware to report all errors to sentry
# This is a server-side middleware that reports any exception from any job
# See more: https://github.com/mperham/sidekiq/wiki/Middleware
module Sidekiq::Middleware::Server
  class SentryErrorLogger
    def call(worker, job, queue)
      begin
        yield
      rescue => error
        Raven.capture_exception(error,
                                extra: {
                                  worker: worker,
                                  job: job,
                                  queue: queue
                                })
        # we raise it after reporting it to sentry.
        # Raise is important here because this raise will tell
        # sidekiq to mark this job as failed and move that job in retry queue
        raise
      end
    end
  end
end

Sidekiq Sentry Middleware

How to use it

Add it in Sidekiq middleware folder and register it in Sidekiq configuration.

# Add SentryErrorLogger to lib/sidekiq/middleware/server/
# Register SentryErrorLogger in `config/initializers/sidekiq.rb`
Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Sidekiq::Middleware::Server::SentryErrorLogger
  end
end

Configuration to register Sidekiq Sentry middleware

It should work out of the box, provided you have Sentry (or any other source where you want to report) configured in your application.

Final Thoughts

After we started using Sidekiq error reporting, we were able to minimize the impact of failing background jobs. Now we have more visibility into our background jobs. When jobs fail we get alerts with the stacktrace, which helps a lot during debugging. Early detection results in an early fix, which in turn means minimal impact on users.