Do YOU know Resque?

by on November 8, 2010

Do YOU know Resque?

This guest post is by Dave Hoover, who authored the book Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman for O’Reilly, instigated the Software Craftsmanship North America conference, and is the Chief Craftsman at Obtiva. Dave began teaching himself to program in 2000, back when he was a family therapist. Dave lives near Chicago with his wife and three children. In his spare time, Dave competes in endurance sports.

Dave Hoover Web developers can sometimes forget the importance of doing as little work as possible during the HTTP request-response life-cycle. When we’re developing new features, the simplest thing to do is just handle all the work that has been requested before responding, making the user wait, patiently watching their browser spin. This is nearly always a bad idea, and for reasons beyond user experience, most notably, it ties up a web process, making your site more likely to experience outages as traffic spikes. While it often makes sense to develop features with slow responses for your initial implementation, it’s usually unwise to deploy that version of the feature to your production environment. Thankfully, Ruby developers can choose from a number of “background job” libraries. I’m going to introduce you to Resque, developed at Github, built on top of Redis, an advanced key-value store which Resque uses for queuing.

One nice thing about Resque is that it’s not dependant on Rails or any web framework. This is great, because today, I’m not interested in writing a web application. I want to write a fast-running Ruby program that figures out what work needs to be done, tells someone else to do it, and then exits. (Similar to a web request, but simpler.) I’ll start with this:

idea = ARGV
puts "Analyzing your idea: #{idea.join(" ")}"
idea.each do |word|
  puts "Asking for a job to analyze: #{word}"
  # This is where we would enqueue something
end

If I named this program idea_analyzer.rb and it was in my current working directory, I could run it like this:

$ ruby idea_analyzer.rb I will learn ruby
Analyzing your idea: I will learn ruby
Asking for a job to analyze: I
Asking for a job to analyze: will
Asking for a job to analyze: learn
Asking for a job to analyze: ruby

As you can see, this program takes an “idea” from the command line and claims to ask for a “job” to analyze each word in the idea. Obviously, the next step is to actually ask for that job, instead of just talking about it. First, I’ll write the code that tells Resque to enqueue a job, and then we’ll get Resque in place. That might seem backward to some of you, since I’m writing code I know will fail, but “fast-failure” is a technique I use all the time, whether I’m practicing test-driven development, or learning a new technology with a toy problem like this:

idea = ARGV
puts "Analyzing your idea: #{idea.join(" ")}"
idea.each do |word|
  puts "Asking for a job to analyze: #{word}"
  Resque.enqueue(WordAnalyzer, word)
end

Nice and simple. We’re calling a method on the Resque class. We’re passing in the word, but we’re also passing in the class WordAnalayzer. This is the only code that interacts directly with Resque. The enqueue method takes the name of the class responsible for doing the background work and the data required to accomplish the work, in this case the word variable. It will attempt to place a job in the appropriate queue. If we run the current version of this program, it fails like this:

$ ruby idea_analyzer.rb I will learn ruby
Analyzing your idea: I will learn ruby
Asking for a job to analyze: I
idea_analyzer.rb:5: uninitialized constant Resque (NameError)
	from /tmp/stuff.rb:3:in `each'
	from /tmp/stuff.rb:3

The uninitialized constant Resque error is telling me that Ruby doesn’t know about Resque yet. I can fix that by installing the Resque gem.

$ gem install resque
Successfully installed resque-1.10.0
1 gem installed
Installing ri documentation for resque-1.10.0...
Installing RDoc documentation for resque-1.10.0...

You’ll likely see other gems being installed as well, these are the gems that Resque depends on. Now I’ll just tell our program about Resque:

require "resque"

idea = ARGV
puts "Analyzing your idea: #{idea.join(" ")}"
idea.each do |word|
  puts "Asking for a job to analyze: #{word}"
  Resque.enqueue(WordAnalyzer, word)
end

When we run this, we’ll get a different error. Excellent! We’re making progress.

$ ruby idea_analyzer.rb I will learn ruby
Analyzing your idea: I will learn ruby
Asking for a job to analyze: I
idea_analyzer.rb:7: uninitialized constant WordAnalyzer (NameError)
	from idea_analyzer.rb:5:in `each'
	from idea_analyzer.rb:5

If you see an error like no such file to load -- resque, then you need to add require "rubygems" at the top of your program. You should eventually see the error about a missing WordAnalyzer. I’ll take care of that next by creating a word_analyzer.rb file, defining the class…

class WordAnalyzer

end

…and then require it.

require "resque"
require "word_analyzer"

idea = ARGV
puts "Analyzing your idea: #{idea.join(" ")}"
idea.each do |word|
  puts "Asking for a job to analyze: #{word}"
  Resque.enqueue(WordAnalyzer, word)
end

And this fails with a different error, we’re almost there!

$ ruby idea_analyzer.rb I will learn ruby
Analyzing your idea: I will learn ruby
Asking for a job to analyze: I
/my/gems/resque-1.10.0/lib/resque/job.rb:44:in `create': Jobs must be placed onto a queue. (Resque::NoQueueError)
	from /my/gems/resque-1.10.0/lib/resque.rb:206:in `enqueue'
	from idea_analyzer.rb:8
	from idea_analyzer.rb:6:in `each'
	from idea_analyzer.rb:6

Now our problem is that we haven’t specified a queue for the WordAnalyzer class. As its name suggests, Resque is all about queues. Each Resque class, such as WordAnalyzer, can specify its default queue, like this:

class WordAnalyzer
  @queue = "word_analysis"
end

Re-running this results in:

$ ruby idea_analyzer.rb I will learn ruby
Analyzing your idea: I will learn ruby
Asking for a job to analyze: I
/my/gems/redis-2.0.10/lib/redis/client.rb:226:in `connect_to': Connection refused - Unable to connect to Redis on localhost:6379 (Errno::ECONNREFUSED)

Resque is trying to enqueue a WordAnalyzer job for “I” on the word_analysis queue, and is using the default host (localhost) and port (6379). I’ll start Redis and our program should be much happier. I recommend installing Redis via https://github.com/antirez/redis/archives/master with antirez-redis-v2.0.3-stable-0-gb766149.zip. Then starting it in a new console with redis-server. With that running, you can rerun your program and it should look like the output of the first version:

$ ruby idea_analyzer.rb I will learn ruby
Analyzing your idea: I will learn ruby
Asking for a job to analyze: I
Asking for a job to analyze: will
Asking for a job to analyze: learn
Asking for a job to analyze: ruby

But this time, after its quick run, it has left some work behind, sitting in Redis. You can see it if you type resque-web in your console. This will launch a browser and bring up a little Sinatra app that ships with Resque, allowing you to watch the activity between Resque’s queues and workers. Now that we can see 4 jobs waiting patiently in the word_analysis queue, let’s get a worker started. The customary way to start Resque workers is via Rake, so I’ll create a Rakefile beside my other 2 files and just put this in the Rakefile:

require "word_analyzer"
require "resque/tasks"

Then, from the command line, I can start the worker with:

rake resque:work QUEUE=*

This will start a worker listening on all of Resque’s queues, and will never exit. If you want to stop it, just hit CTRL-C. Once it has run for a few seconds, refresh the browser you had pointing at resque-web, click-through to the failure queue, and you’ll see all the jobs failed with undefined method 'perform' for WordAnalyzer:Class. That’s a nice way of telling us it’s time to write the perform method for our Resque class:

class WordAnalyzer
  @queue = "word_analysis"

  def self.perform(word)
    puts "About to do heavy duty analysis on #{word}"
    sleep 3 # fake analysis here
    # this would be something impressive
    puts "Finished with analysis on #{word}"
  end
end

If your worker is still running, stop it with a CTRL-C. Then restart it via Rake so it loads up our new perform method. As you’ve probably guessed, Resque simply calls a method named perform on the class you enqueue. Be aware that any arguments you pass into Resque.enqueue are going to be serialized as JSON, which means Ruby Symbols will turn into Strings, and complex objects like instances of ActiveRecord will not work. When I need to work with ActiveRecords in Resque, I just pass their ids across and re-query them from the database.

Now that the worker is restarted and WordAnalyzer knows what to perform, our background processing system is ready. Start a new console and execute ruby idea_analyzer.rb I will learn ruby. Your Resque worker should perform some “successful” analysis over the course of about 12 seconds:

$ rake resque:work QUEUE=*
(in /Users/redsquirrel/Desktop)
About to do heavy duty analysis on I
Finished with analysis on I
About to do heavy duty analysis on will
Finished with analysis on will
About to do heavy duty analysis on learn
Finished with analysis on learn
About to do heavy duty analysis on ruby
Finished with analysis on ruby

That’s all there is to it. You can keep running your idea_analyzer.rb and the worker will keep analyzing words. Here is a visual workflow of this little system that may help clarify the different roles:

resque_workflow

We have a fast running program that queues work for later. We have a simple class that performs a time-consuming job, managed by a long-running Resque worker. We also have a web interface to monitor our queues and workers. These are the building blocks used by large-scale web sites like Github, Mad Mimi, and Groupon, who leverage Resque for their mission critical background processing.

I hope you found this article valuable. Feel free to ask questions and give feedback in the comments section of this post. Thanks!

Do also read these awesome Guest Posts:

Technorati Tags: , ,

Posted by Dave Hoover

Follow me on Twitter to communicate and stay connected

{ 46 comments… read them below or add one }

Dani November 8, 2010 at 1:55 pm

Very simple and clear. Thanks for this introduction to Resque.

Reply

Nícolas Iensen November 8, 2010 at 3:51 pm

Thanks for the well explained topic!
There is a way to check if the workers are down automatically?

Reply

Dave Hoover November 8, 2010 at 7:09 pm

Resque doesn’t ship with any worker monitoring tools. It’s best to use God or Monit to manage workers.

Reply

Rick November 9, 2010 at 11:17 am

does this setup work with windows?

Reply

Dave Hoover November 9, 2010 at 9:00 pm

Redis would have trouble on windows, though it does look like people have got it working with cygwin: http://code.google.com/p/redis/issues/detail?id=34

Reply

vr January 7, 2011 at 2:58 am

when i do follow your example,everything worked except Sinatra app,
when i do this
$ resque-web

i get following error,any help will be apreciated
/Library/Ruby/Gems/1.8/gems/rack-1.2.1/lib/rack/utils.rb:138:in `union’: can’t convert Array into String (TypeError)
from /Library/Ruby/Gems/1.8/gems/rack-1.2.1/lib/rack/utils.rb:138
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `gem_original_require’
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `require’
from /Library/Ruby/Gems/1.8/gems/rack-1.2.1/lib/rack/request.rb:1
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `gem_original_require’
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `require’
from /Library/Ruby/Gems/1.8/gems/rack-1.2.1/lib/rack/showexceptions.rb:3
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `gem_original_require’
… 9 levels…
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `require’
from /Library/Ruby/Gems/1.8/gems/resque-1.10.0/bin/resque-web:10
from /usr/bin/resque-web:19:in `load’
from /usr/bin/resque-web:19

Reply

Dave Hoover January 8, 2011 at 7:27 pm
Karmen Blake January 12, 2011 at 6:21 am

great article. very clear!

Reply

vr January 13, 2011 at 1:54 am

am trying to use resque-request,it lets me queue the requests but when the worker is ran,it givies “The task failed because of an error: uninitialized constant” for helper or model we use.do you have any idea on how to load helper and models into worker class.

Thanks.

Reply

Dave Hoover January 13, 2011 at 10:27 am

I could help you if I could see the code and the full error. Please post it to https://gist.github.com and reply with the URL.

Reply

vr January 14, 2011 at 6:06 am

got it working,it was not loading rails environment.Thanks for all this article,it helped to get me started.

Reply

Leddo January 21, 2011 at 10:42 pm

What did you do to get it to load the Rails Environment? I’m getting the same problem.

Reply

vr January 24, 2011 at 8:03 pm

put

task(“resque:setup” => :environment)

in rake file

Reply

Dave Hoover January 22, 2011 at 9:04 am

To get rake to load the Rails environment, you execute it with: rake environment resque:work QUEUE=*

See https://github.com/defunkt/resque and search for “environment”

Reply

yacobus reinhart January 24, 2011 at 3:27 am

I have run all your instructions, without any error. at final step after i run Rake, i execute this:

developer@T****sis:~/stream$ sudo ruby test_bg.rb I believe I can fly
Analyzing your idea: I believe I can fly
Asking for a job to analyze: I
Asking for a job to analyze: believe
Asking for a job to analyze: I
Asking for a job to analyze: can
Asking for a job to analyze: fly
developer@T****sis:~/stream$

My question, why no worker is working for my queues? i wait more than 20 minutes, and still not executed.

Reply

Dave Hoover January 24, 2011 at 9:47 am

Do you have a worker processing running? If it’s not still running, execute: rake resque:work QUEUE=*

Reply

yacobus reinhart January 24, 2011 at 4:41 pm

I did execute rake resque:work QUEUE=*

i am using passenger-apache and REE Rails 3

Reply

yacobus reinhart January 24, 2011 at 9:07 pm

I can’t get my worker’s working but the pid is registered in stats/keys.

I have put
task(“resque:setup” => :environment)
in my Rakefile, and restarting passenger.

I always get stuck here:
developer@T****sis:~/stream$ rake environment resque:work QUEUE=*
(in /home/developer/stream)
[No information about worker like this tutorial after line above]

Theses are some information about my configuration:
File: config/initializers/resque.rb
require ‘resque’

redis_config = YAML.load_file(“#{Rails.root}/config/redis.yml”)
Resque.redis = redis_config[Rails.env]
Resque.redis.namespace = “resque:company:namespace”

File: config/redis.yml
development: localhost:6379
test: localhost:6379
staging: localhost:6379
production: localhost:6379

what did i do wrong, so that my worker doesn’t work? but the pid is registered.

Reply

Dave Hoover January 25, 2011 at 10:12 am

This part is expected and totally fine:
[No information about worker like this tutorial after line above]

The worker runs forever, pulling jobs off of queues. You need to keep it running in order for Resque jobs to be processed.

yacobus reinhart January 24, 2011 at 10:07 pm

it works excellent now :), thanks for this page.
the problem is in namespace of config/initializers/resque.rb
specially here :
Resque.redis.namespace = “resque:company:namespace”

you should describe your namespace like:
sudo resque-web -N resque:company:namespace

so you will get the right worker work based on name space, if you have different namespace the worker will be allocated to your described namespace.

Reply

Fred February 20, 2011 at 1:42 pm

Very helpful. What’s the different with work and workers in rake -T tasks? I think that the work is enough.

Reply

yacobus reinhart February 22, 2011 at 8:33 am

Work is for 1 worker but workers is more than 1, if you have multiple tasks, using workers will be very helpful and much better if you can make dynamic workers.

Reply

Eric Marden September 21, 2011 at 4:09 am

The ‘workers’ task allows you to start multiple workers, by passing in an environment variable representing how many you want. You can even specify the queue you want them to work on, or all of them if you pass in a ‘splat’.

Here’s twenty workers which will work on any queue:

`COUNT=20 QUEUE=* bundle exec rake resque:workers`

Reply

Tom August 4, 2011 at 6:49 pm

Great article. Short and sweet.

Reply

yatish October 21, 2011 at 1:49 pm

how can we alot more same workers for a single queue.

Reply

Dave Hoover October 21, 2011 at 6:07 pm

In development mode, you can set a COUNT environment variable to tell Resque how many workers to start.

COUNT=5 QUEUE=some_queue_name rake resque:workers

Reply

Chris olido November 9, 2011 at 3:00 pm

is there a way i can parse the resque job to a collectd tool like so i can have the metrics of my jobs?

Reply

Dave Hoover November 9, 2011 at 6:29 pm
yatish November 9, 2011 at 7:57 pm

when i use the env variable COUNT=5 then 5 workers should work parallely but in my case only one worker is working when it finishes the job then it takes the next job. 5 workers shud work parallely that not happening.

Reply

Jeronimo February 8, 2012 at 8:55 pm

Very good introduction to Resque and still valid in 2012. I was having a lot of doubts on how to start and your post gives a sequential learning on the topic
Congratulations.

Reply

binod June 19, 2012 at 7:40 pm

i had to do this for the require word_analyzer to work

require “./word_analyzer”

Reply

Dave Hoover June 19, 2012 at 8:26 pm

Yes, you would need to do that in order for it to work on Ruby 1.9.2+

Reply

Cristopher Araya August 24, 2012 at 8:45 pm

Nice !!
Thank you so much !

Reply

Ivan August 31, 2012 at 10:12 pm

Excellent resource!
A great, straightforward tutorial to give us developers the gist of Resque-knowledge.

Reply

Joel December 7, 2012 at 2:38 pm

Thanks for this great intro to ‘Resque’…………

Reply

Malak December 14, 2012 at 10:07 pm

Simple and effective – great tutorial. Thanks.

Reply

yatish December 15, 2012 at 11:24 am

Try sidekiq instead. Its similar to resque but very less RAM and awesome performance

Reply

ping li January 22, 2013 at 11:42 am

I am confused with the relations between worker and (QUEUE|@queue). If one worker is to start a job in the QUEUE which name(@queue) is the same as the worker name.

Reply

Jeff Chen March 20, 2013 at 2:12 am

Hi got to the last step (running in windows).
and got the following warning:

rake resque:work QUEUE=*
WARNING: This way of doing signal handling is now deprecated. Please see http://
hone.heroku.com/resque/2012/08/21/resque-signals.html for more info.

it was removed from the queue, but i didn’t see the print out for:
puts “Finished with analysis on #{word}”
and also it exited immediately after.

Any ideas what i did wrong?

Reply

Asish March 28, 2013 at 10:40 pm

nice article… but… I am facing some trouble while i put task in background and try to start the queue.
resque overview page shows that the task has been sent to the worker queue, but when i try to start the queue with rake resque:work QUEUE=’*’ it gives the following error
Loading Rails environment for Resque
Initializing connection to redis…
Done (port = 6392)!
Configuring resque sheduler…
Resque scheduler configured
rake aborted!
database configuration does not specify adapter
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract/connection_specification.rb:47:in `resolve_hash_connection’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract/connection_specification.rb:41:in `resolve_string_connection’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract/connection_specification.rb:27:in `spec’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract/connection_specification.rb:130:in `establish_connection’
/home/woody/Workspace/IDP/Amigos/app/models/ak.rb:2
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:469:in `load’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:469:in `load_file’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:639:in `new_constants_in’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:468:in `load_file’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:353:in `require_or_load’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:313:in `depend_on’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:225:in `require_dependency’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/railties-3.2.13/lib/rails/engine.rb:439:in `eager_load!’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/railties-3.2.13/lib/rails/engine.rb:438:in `each’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/railties-3.2.13/lib/rails/engine.rb:438:in `eager_load!’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/railties-3.2.13/lib/rails/engine.rb:436:in `each’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/railties-3.2.13/lib/rails/engine.rb:436:in `eager_load!’
/home/woody/.rvm/gems/ruby-1.8.7-p371/gems/resque-1.24.1/lib/resque/tasks.rb:61
/home/woody/.rvm/gems/ruby-1.8.7-p371/bin/ruby_noexec_wrapper:14
Tasks: TOP => resque:work => resque:preload
(See full trace by running task with –trace)

database connection seems okay… because i get list of data in the index page.
and i can see queued tasks in the overview page (http://localhost:3000/resque/overview)

Could you please please please help me?

Reply

Satish May 8, 2013 at 4:11 pm

It was a headache to change in working code of redis. Thanks a lot your article helps me to understand how to play with redis.

Reply

Akash Khandelwal May 21, 2013 at 1:17 pm

After the step of requiring “word_analyzer” in idea_analyzer.rb and running that file I am getting the error:

You are using an old or stdlib version of json gem
Please upgrade to the recent version by adding this to your Gemfile:

gem ‘json’, ‘~> 1.7.7′

/home/akash/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file — word_analyzer (LoadError)
from /home/akash/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from idea_analyzer.rb:2:in `’

However, I have added this gem of same version.

Reply

Patrick J. Sparrow May 21, 2013 at 8:08 pm

“… arguments you pass into Resque.enqueue are going to be serialized as JSON, which means Ruby Symbols will turn into Strings, and complex objects like instances of ActiveRecord will not work.”

This is very important to remember, and if you use mocks in your tests, you may not catch it.

Reply

aashish May 31, 2013 at 3:53 pm

Hi,
How do allocate one particular queue to specific worker.
Let us assume i have 5 queues and 5 workers
Can i expect something like 3rd queue running on worker=1 and 5th queue running on worker=4.
Is it possible!! as this
bundle exec rake resque:work QUEUE=3 WORKER=1

Thank you,
Aashish

Reply

Patrick J. Sparrow May 31, 2013 at 4:40 pm

@Aashish,

You specify the name of the queue inside your workers. In the example below, MyWorker will only process items from queue1 and MyOtherWorker will only process items from queue2.

class MyWorker
@queue = “queue1″
end

class MyOtherWorker
@queue = “queue2″
end

Reply

asihy May 31, 2013 at 5:13 pm

How do specify on heroku. Process 2 queues parallelly on 2 heroku workers.

Reply

Leave a Comment

{ 68 trackbacks }

Previous post:

Next post: