Performance Testing Rails Applications — How To?

This guest post is by Gonçalo Silva, who is a full-time Ruby on Rails developer at escolinhas.pt and has participated in the Ruby Summer of Code 2010. He loves and contributes to many open-source projects, being a fan of Linux, Ruby and Android. He likes to call himself a hacker, but that’s just an excuse for being in front of the computer all the time. Oh, and he tweets at @goncalossilva.

Rails 3.1 is just around the corner, and it brings enhanced performance testing tools. Let’s have a look at this often overlooked feature of our web application framework of choice.

This isn’t new

Rails has had built-in performance testing tools since version 2.2. Originally developed by Jeremy Kemper, these allowed developers to test the performance of their applications by writing integration tests which could be benchmarked and profiled under MRI. He later introduced two scripts – benchmarker and profiler – which were great to quickly benchmark or profile small snippets of code.

Actually, this is kind of new

I came across these tools during last year’s Ruby Summer of Code. I remember feeling astonished and bit ashamed about not having played with them before. I couldn’t use them at their full potential because of the lack of full support for YARV (or MRI 1.9), so I set off fixing that. While working on it, I’ve made a list of other things these tools lacked, that I wanted to implement after RSoC, namely: – Rubinius support – JRuby support – Test configurability – Decoupling benchmarker and profiler from RubyProf

Everything listed above is now implemented. Rails 3.1 will ship with these improvements and we’ll no longer have excuses for not using these great tools Rails provides for all of us.

Why you should care

The web should be fast. Response times are a key factor in user experience and there is very limited patience for slow websites. Ruby interpreters aren’t famous for being performant and our beloved framework isn’t known for getting faster with new releases. Nevertheless, we want our websites to be fast and responsive, and buying tons of hardware isn’t always an available choice – we need our code to be fast. We should care.

How does this work?

Rails’ performance testing tools allow you to quickly detect performance bottlenecks. As a rule of thumb, use benchmarking to detect the problem and then use profiling to understand it. Profiling provides in-depth information about your code and what it’s doing, but it lacks the speed and simplicity of benchmarking.

Patching your Ruby interpreter

You can skip this section if you’re a Rubinius/JRuby/REE user.

If you’re an MRI/YARV user, you’ll need a patched interpreter to access all available metrics. Before you run off, let me tell you that it’s very simple to install a patched Ruby interpreter nowadays. Thanks to Wayne, the author of RVM, all you need to do is to specify an additional flag when installing your interpreter, like this: rvm install 1.9.2 --patch gcdata Or, if you’re still using 1.8 (really?): rvm install 1.8.7 --patch ruby187gc

That’s all, folks. You now have a patched Ruby interpreter. If you want, you can have your patched interpreter side by side with your regular one, by simply assigning a name to it:

rvm install 1.9.2 --patch gcdata --name perf
rvm 1.9.2-perf  # my patched interpreter
rvm 1.9.2       # my regular interpreter

And that’s it.

Editing your Gemfile

You can skip this section if you’re using Rubinius/JRuby.

If you’re not, you’ll need to add RubyProf to your Gemfile:

gem 'ruby-prof', :git => 'git://github.com/wycats/ruby-prof.git'

Don’t forget to remove this from your Gemfile and re-run bundle install if you intend to switch to Rubinius or JRuby.

Performance tests

In order to use these tools, you’ll need to write performance tests. These tests are just like integration tests, except that the point is not to assert anything. They’ll just run the code that you want to see benchmarked/profiled.

Generating

As expected, Rails does this stuff for you. Just run:

script/rails generate performance_test example

And a new file will be placed in test/performance/example_test.rb containing the default test:

require 'test_helper'
require 'rails/performance_test_help'
class ExampleTest < ActionDispatch::PerformanceTest
  # Refer to the documentation for all available options
  # self.profile_options = { :runs => 5, :metrics => [:wall_time, :memory]
  #                          :output => 'tmp/performance', :formats => [:flat] }

  def test_homepage
    get '/'
  end
end

Editing

Since ActionDispatch::PerformanceTest inherits from ActionDispatch::IntegrationTest, you can use all available helpers for integration tests in your performance tests. For instance, if you wanted a test for your login action you could use:

class LoginTest < ActionDispatch::PerformanceTest
  fixtures :users
  self.profile_options = { :metrics => [:wall_time, :memory] }

  def test_login
    post_via_redirect "/login", :username => users(:youruser).username, :password => users(:youruser).password
  end
end

Tweaking

Starting with Rails 3.1, performance tests can be configured. As you’ve probably figured out from the aforeshown LoginTest, all you need to do is to specify an optional hash of options to use when benchmarking/profiling. You can use one set of options for each class. Not all options are available to all interpreters, especially the ones related with profiling. Metric/output availability for each interpreter will be shown below. You can skip this section and come back later, after grasping the whole concept. You’ll also be able to check it out on Rails’ performance testing guide once 3.1 comes out.

Metric availability

Benchmarking

Profiling

Output availability

Running

Finally, it’s time to run your tests. Let’s start with benchmarking:

rake test:benchmark

And the output should be similar to this:

ExampleTest:
ExampleTest#test_homepage (16 ms warmup)
           wall_time: 0 ms
              memory: 17 KB
             objects: 195
             gc_runs: 0
             gc_time: 0 ms
 homepage (0.75s)

LoginTest:
LoginTest#test_login (92 ms warmup)
           wall_time: 10 ms
              memory: 180 KB
 login (0.44s)

Finished in 1.193759 seconds.

If any result disappoints you, profile it:

rake test:profile TEST=test/performance/login_test.rb

And you should get a similar output:

LoginTest:
LoginTest#test_login (105 ms warmup)
           wall_time: 69 ms
              memory: 2.4 KB
 login (5.02s)

Profiling will give you much more information than what’s printed on your terminal.

Reviewing results

By default, performance tests store their results in tmp/performance (although it can be changed by specifying a value for :output in the profile_options hash). For benchmarks, this is pretty straightforward: it stores one CSV per metric (LoginTest#test_login_memory.csv, for instance) with the results as time goes by.

measurement,created_at,app,rails,ruby,platform
183222,2011-08-10T18:15:09Z,,3.1.0.rc5,ruby-1.9.2.290,i686-linux
216344,2011-08-11T14:37:59Z,,3.1.0.rc5,ruby-1.9.2.290,i686-linux
(...)

When profiling, however, the result files are extremely important. They contain the juicy details of your test runs. Similarly to benchmarking results, there will be one file per metric. There are, however, multiple formats available, specially if you’re using RubyProf (and consequently MRI/REE/YARV). These formats can range from messy flat text files to awesome HTML stack traces, and they will provide valuable input when spotting bottlenecks.

The scope of this article is not to explore RubyProf’s available output formats, but you should have a look at the available printers. However, keep in mind that RubyProf supports more metrics and output formats than Rubinius/JRuby‘s profilers. These can only measure wall time when profiling, and will only print their results in Flat/Graph text formats.

RubyProf’s HTML stack printer

Quick tests

Performance tests are great, but they can be inconvenient when all you want is to quickly test a small snippet of code. For this, Rails provides two command line tools: benchmarker and profiler.

Open your terminal and run:

rails benchmarker 'User.all'

And it will work as if you had created a performance test and put that code in it. Very simple, right? Another example:

rails profiler 'User.all' 'User.find_by_login("goncalossilva")' --runs 3 --metrics cpu_time,memory # profiling memory won't work under Rubinius/JRuby (benchmarking memory will!)

Two things pop up from this code snippet: you can run multiple tests in a single command and you can specify options as you would with normal performance tests.

To get a glimpse at all available options, run:

rails benchmarker --help rails profiler --help

What can be done with this?

A lot of things can be accomplished with these tools. First and foremost, you can assess the performance of your application by benchmarking certain parts, either through tests or simple snippets of code. After finding potential bottlenecks, you can use profiling to gain a greater insight into what’s happening and how it can be improved.

There are other useful tasks that can be done with these tools. You could, for instance, compare the performance of different interpreters on your application:

    rvm 1.9.2
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory
    rvm ree
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory
    rvm rubinius
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory
    rvm jruby
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory

Now you’ll know which interpreter takes less/more time/memory when it’s opening your homepage/running MyModel.slow_method.

Giving it a try

If you’ve come this far, now you know how to use these powerful tools. Try playing with them: I’m sure you’ll find valuable information about your applications’ performance, and potentially spot some easily fixable bottlenecks. With little effort, your application will be faster, you will be prouder and your users will be happier!

Feel free to ask questions and give feedback in the comments section of this post. Gonçalo has also written a guest blog post for RubyLearning before, titled – “Ruby gems — what, why and how“. Fellow Rubyists, if you would like to write a guest blog post for RubyLearning write to satish [at] rubylearning.org

Ruby gems — what, why and how

This guest post is by Gonçalo Silva, who is a full-time Ruby on Rails developer at escolinhas.pt and has participated in the Ruby Summer of Code 2010. He loves and contributes to many open-source projects, being a fan of Linux, Ruby and Android. He likes to call himself a hacker, but that’s just an excuse for being in front of the computer all the time. Oh, and he tweets at @goncalossilva.

What is a gem

At its most basic form, a Ruby gem is a package. It has the necessary files and information for being installed on the system. Quoting RubyGems: «A gem is a packaged Ruby application or library. It has a name (e.g. rake) and a version (e.g. 0.4.16)».

Being very powerful, gems are of great importance in the Rubyland. They can easily be used to extend or change functionality within Ruby applications.

Structure

Every gem is different, but most follow a basic structure:

gem/
|-- lib/
|   |-- gem.rb
|-- test/
|-- README
|-- Rakefile
|-- gem.gemspec

Your gem’s code is located under lib/ which typically holds a Ruby file with the name of the gem. You can choose to have all the magic happening in this file, but you can also use it to load some other Ruby files also located under lib/, typically inside a folder with the gem’s name. Confused? Have a look:

your_gem/
|-- lib/
|   |-- your_gem.rb
|   |-- your_gem/
|   |   |-- source1.rb
|   |   |-- source2.rb
|-- ...

The test folder’s name is not necessarily named test/. When you’re working with RSpec, for instance, its name is usually spec/. As you’ve probably guessed, this folder holds tests for your gem.

After the README file, which hopefully doesn’t need any introduction, comes the Rakefile. In a gem’s context, the Rakefile is extremely useful. It can hold various tasks to help building, testing and debugging your gem, among all other things that you might find useful.

The gemspec—as the name implies—contains your gem’s specification by defining several attributes. An example gemspec file could be:

Gem::Specification.new do |s|
  s.name              = "gem"
  s.version           = "0.0.1"
  s.platform          = Gem::Platform::RUBY
  s.authors           = ["Gonçalo Silva"]
  s.email             = ["goncalossilva@gmail.com"]
  s.homepage          = "http://github.com/goncalossilva/gem_template"
  s.summary           = "Sample gem"
  s.description       = "A gem template"
  s.rubyforge_project = s.name

  s.required_rubygems_version = ">= 1.3.6"

  # If you have runtime dependencies, add them here
  # s.add_runtime_dependency "other", "~> 1.2"

  # If you have development dependencies, add them here
  # s.add_development_dependency "another", "= 0.9"

  # The list of files to be contained in the gem
  s.files         = `git ls-files`.split("\n")
  # s.executables   = `git ls-files`.split("\n").map{|f| f =~ /^bin\/(.*)/ ? $1 : nil}.compact
  # s.extensions    = `git ls-files ext/extconf.rb`.split("\n")

  s.require_path = 'lib'

  # For C extensions
  # s.extensions = "ext/extconf.rb"
end

Some attributes like the name, version, platform and summary are required others are optional. If you use git with your project, you can use the nifty trick shown above to list the project’s files, executables and extensions. If you don’t, you can simply fall back to using pure Ruby code like:

s.files = Dir["{lib}/**/*.rb", "{lib}/**/*.rake", "{lib}/**/*.yml", "LICENSE", "*.md"]

The dummy gem is very simple. Because of this, it perfectly illustrates some of the ideas explained above. Some interesting bits are shown below:

dummy/
|-- lib/
|   |-- dummy/
|   |   |-- core_ext/
|   |   |   |-- array.rb
|   |   |   |-- string.rb
|   |   |-- address.rb
|   |   |-- company.rb
|   |   |-- ...
|   |-- dummy.rb

This gem is organized into several source files inside lib/. The dummy.rb implements the top-level module and loads all functionality from the Ruby files inside lib/dummy/. It also includes some core extensions, namely to the Array and String classes (which are part of Ruby’s core).

RubyGems

Finally, RubyGems. It is a package manager which became part of the standard library in Ruby 1.9. It allows developers to search, install and build gems, among other features. All of this is done by using the gem command-line utility. You can find its website at rubygems.org.

Why is this useful

Gems are very useful for not reinventing the wheel and avoiding duplication. That’s basically it. Many Ruby developers create and publish awesome gems which address specific requirements, solve specific problems or add specific functionality. Anyone who comes across similar requirements or problems can use them and eventually improve them. That’s the joint awesomeness of Ruby’s strong open-source foundation and extreme flexibility. Anyway, you’re reading this article… so you’ve probably understood the concept and grasped its usefulness long before reading this paragraph.

How to make your own

Making your own gem is nothing more than packaging your library or application according to the structure stated above. Put all your code under lib/, all your tests under test/ or spec/, your gem specification under your_gem.gemspec and you’re good to go. Of course, a few other files might come in handy, namely a Rakefile, a README and a LICENSE. A CHANGELOG, sometimes, might be useful as well.

Ruby Idioms

When developing a gem, you are probably creating, extending or overriding functionality. You might want people to include your module in their classes, or perhaps you just want to extend a given class with your module—it’s your choice. What you shouldn’t really do, however, is reinventing Ruby’s module system. There is an excellent blog post on this which can help if you—like many gem authors I’ve seen—start overriding include to behave like extend. It’s very important to understand the difference between the two and, fortunately, there are great resources about this out there.

Developing with Bundler

Using Bundler to manage your gem’s dependencies is also pretty easy. Just create a Gemfile and add:

gemspec

After this, fire Bundler:

bundle install

And yes, you got it right. After adding gemspec to your Gemfile, Bundler can scan your gemspec, find your runtime and development dependencies and install them for you.

While not being mandatory, I strongly recommend you to consider using Bundler to manage your gem’s dependencies. If used correctly, it can probably be a time saver.

Testing

When it comes to testing, you’ve got plenty of good options. Some people rely on test-unit (or minitest in 1.9), others prefer RSpec. It’s really up to you. The only bad choice you can possibly make is opting to not testing your gems at all.

Once again, I’m going to use dummy’s simplicity to explain this a bit further. All tests were built on test-unit and are organized as follows:

dummy/
|-- test/
|   |-- address_test.rb
|   |-- company_test.rb
|   |-- ...
|   |-- test_helper.rb

As you’ve seen, tests are structured similarly to dummy itself. The test_helper is in charge of loading the necessary libraries and setting up any variables or methods used across most (if not all) tests. All tests are organized into files which target specific functionality in dummy. The tests contained in address_test.rb run against address.rb and so on.

Publishing

After everything is coded and tested, all you got left to do is packaging and publishing. The previously mentioned gem utility makes it all very simple. Just run gem build your_gem.gemspec and you should see something along these lines:

Successfully built RubyGem
Name: your_gem
Version: 0.0.1
File: your_gem-0.0.1.gem

Pushing your gem to RubyGems is as easy as it is to build it. Just gem push your_gem-0.0.1.gem and soon it’ll be published. Be aware that the first time you issue this command you’ll be prompted to login with a RubyGems.org account.

Concerning this, I like keeping these simple tasks in my Rakefile:

desc "Validate the gemspec"
task :gemspec do
  gemspec.validate
end

desc "Build gem locally"
task :build => :gemspec do
  system "gem build #{gemspec.name}.gemspec"
  FileUtils.mkdir_p "pkg"
  FileUtils.mv "#{gemspec.name}-#{gemspec.version}.gem", "pkg"
end

desc "Install gem locally"
task :install => :build do
  system "gem install pkg/#{gemspec.name}-#{gemspec.version}"
end

These help me build and install my gems. They also aid at keeping all packages in the pkg/ folder, which is useful for keeping the root directory clean and tidy.

Gems for building gems

There are a few gems which were specifically created to help developers build their own gems. Among them are the renowned jeweler, hoe and echoe. I can’t go into detail in any of these since I’ve never really used them – I started building my gem skeleton from scratch right at the beginning. However, some of these tools are very helpful so you should really take a look and see if any fits your needs.

Gem template

As I mentioned, I’ve been using a gem skeleton for some time now, which you can find at GitHub. Every gem I’ve built started with that template, which I kept trying to improve over time.

You can start your gems from scratch, but that’s just nonsense. You should create your own skeleton, use one made by someone else or use a third-party gem to help creating your gem.

Legen—wait for it—dary

Ruby gems are filled with awesomeness. Hop in and start making your own!

Feel free to ask questions and give feedback in the comments section of this post. Thanks and Good Luck!