adam bondurant

I ran into a problem with my BackgrounDRb scheduler last night: The MySQL server had to be restarted, and the scheduler got confused because its connection had gone away. By default, a BackgrounDRb worker will not reconnect to its database unless you explicitly tell it to do so. Here's how you can do that.

I found a very helpful post at The Web and All That Jazz that solved my problem. The gist is that Rails will raise an ActiveRecord::StatementInvalid exception if the database server cannot be found. If you rescue this exception, you can add a bit of code to get ActiveRecord to reconnect to your database.

In my case, the issue was not that the database was down for an extended period of time (we have other monitoring scripts to deal with that), but more that when the database did go down, unless we restarted the BackgrounDRb process, it would have a handle to a stale database session, and we just needed to re-fetch the connection handle. Here's the code:

class PersonWorker < BackgrounDRb::Worker::RailsBase
  def initialize
    @retried = false
  end
  
  def do_work
    people = Person.find(:all)
  rescue ActiveRecord::StatementInvalid
    # Our database connection has gone away, reconnect and retry this method
    ActiveRecord::Base.connection.reconnect!
    
    unless @retried
      @retried = true
      retry
    end

    # We've tried to connect twice but the server is still down, re-raise the
    # original exception and get out of here, to avoid a possible infinite loop
    raise
  end
end

There are a few things to look at here: First, the method to reconnect your Rails session to your database is ActiveRecord::Base.connection.reconnect!. If you're connecting to multiple databases, your code is a bit different; if you're using an abstract model class with a call to ActiveRecord::Base.establish_connection, you can just call the reconnect! method on that abstract model's connection object (e.g., MyOtherDatabaseBaseModel.connection.reconnect!). Second, the retry statement tells Ruby to restart an entire block of code or loop (in this case, it goes back to the beginning of the do_work method). Finally, this method only tries reconnecting to your database once. The reason I do this is because this particular worker gets called every 2 minutes, so it's essentially retrying pretty frequently anyway in case MySQL is, in fact, down for a noticeable amount of time. If you need to try reconnecting more frequently, you're on your own for now. (Sorry!)