How do I keep multiple Ruby projects separate?

by on December 20, 2010

How do I keep multiple Ruby projects separate?

This guest post is by Steve Klabnik, who is a software craftsman, writer, and former startup CTO. Steve tries to keep his Ruby consulting hours down so that he can focus on maintaining Hackety Hack and being a core member of Team Shoes, as well as writing regularly for multiple blogs.

Steve Klabnik If you’re anything like me, you’re already starting a new project immediately after wrapping up the last one. There just aren’t enough hours in the day to code up all the crazy ideas I have floating around in my head. Often, these ideas are the result of checking out some fun new gem, GitHub project, or even a different Ruby. Real quickly, a problem develops: what happens when these projects interfere with one another? What if I want to use Ruby 1.8.7 for an older project, Ruby 1.8.5 for a legacy application, Ruby 1.9.2 for the latest and greatest, and JRuby to use an interesting Ruby library? Luckily, there are a few things that you can do to isolate your different projects from one another, and some settings for that will make them quite painless to use. There are three main things that can go wrong when you try to use different sets of tools on a per-project basis: conflicts between Ruby versions, conflicts between gems, and forgetting which tools you use on which project.

Ruby Version Conflicts

This is the biggest and most painful kind of problem. If you want to use Ruby 1.8 for one project and Ruby 1.9 for another, you have a problem. If you’re using Linux, for example, your package manager may see that both ruby18 and ruby19 fulfill a ‘ruby’ dependency, and so it won’t let you have them both installed side by side. The solution isn’t pretty: install different Rubies from source. This gets ugly really quickly, because it’s easy to forget where you’ve compiled different Rubies, and having software outside of your package manager isn’t a great answer. If you’re on OS X or Windows, you skip right past the package manager problem and straight to the source ‘solution.’ This is no good!

Luckily, there’s an awesome project by Wayne E. Seguin named rvm. rvm is sort of like a package manager for Ruby. If you’d like to install both Ruby 1.8.7 and 1.9.2, just type this in:

$ rvm install 1.8.7
$ rvm install 1.9.2

It’ll go fetch the Ruby source code, compile it, and get you all set up. To use a specific Ruby, you can type ‘use’:

$ rvm use 1.8.7
$ ruby -v
ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10.4.0]
$ rvm use 1.9.2
$ ruby -v
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.4.0]

Neat! You can even get other Ruby versions:

$ rvm install jruby
$ rvm install rbx
$ rvm install macruby

You can see a full list of these with ‘rvm list known’. For a full list of everything that rvm can do, as well as installation instructions, visit the rvm website.

Gem Conflicts

Once you’ve gotten your Rubies straight, you can still have conflicts between different gems that your project needs. One project uses Rails 2.3.8, another uses Rails 3… It gets worse when you have certain gems installed only as a dependency, and you don’t know exactly which one is correct:

$ gem list | grep net-ssh
net-ssh (2.0.23, 2.0.4, 1.1.4)

rvm has a neat feature called ‘gemsets.’ They let you create separate sets of gems per Ruby you have installed. This allows you to isolate each application, giving it its own set of gems. Check it out:

$ gem list

*** LOCAL GEMS ***

aasm (2.1.5)
abstract (1.0.0)
acl9 (0.12.0)
*snip*

$ rvm gemset create new-gemset
$ rvm use 1.9.2@new-gemset
$ gem list

*** LOCAL GEMS ***

$

Cool stuff! As you can see, use an ‘@’ symbol to tell rvm which gemset you’d like to use. Now we’ve isolated each project’s gems from each other. There is, however, a much more complicated kind of conflicts that can occur between gems. This happens when two gems have interlocking dependencies.

Here’s an example of this from the past: ActionPack 2.3.5 requires Rack =1.0.0, which is the newest version. Unicorn requires Rack >1.0.0. Rack releases a new version, 1.1.0. Now, when starting up a Rails application, the unicorn gem is loaded first, so it loads the newest version of the gem that works, which is rack-1.1.0. Then rails loads, and it loads actionpack, which tries to load rack. It needs =1.0.1, but sees that 1.1.0 has already been loaded, and throws this ugly, ugly error:

Gem::LoadError: can't activate rack (~> 1.0.0, runtime)
for ["actionpack-2.3.5"], already activated rack-1.1.0
for ["unicorn"]

There’s a set of versions here that works, but the way that the gems are loaded means that it doesn’t. The problem is that at the time that unicorn loads, it can’t possibly know that you’re planning on loading a different version of rack somewhere down the line. What we really need is a tool that knows about all of our dependencies, and can calculate the graph of all of our requirements, and figure out which versions of everything we need, and then only place those versions on the $LOAD_PATH. Luckily, such a project exists: bundler.

To use bundler, you first need to make a file named ‘Gemfile’ in the root of your project directory. This file looks something like this:

source "http://rubygems.org"

gem "rails", "~>3.0.0"

group :development do
  gem 'sqlite3-ruby', :require => 'sqlite3'
end

group :production do
  gem "pg"
end

The first line tells Bundler where to look for gems. The second line says that we want to use the ‘rails’ gem, and we want any version that’s at least 3.0.0 but less than 3.1.0. Finally, the other lines show ‘groups’ of gems: in development, we want to use sqlite3-ruby, and we need to require it via the name ‘sqlite3′, but we want to use Postgres in production. To install these gems, just:

$ bundle install

Bundler gets all the information that it needs on all the gems, figures out what versions of everything work together, and then installs the right versions. It then creates a Gemfile.lock file that holds all of this information. It’s just a simple YAML file, you can open it up and see the specifics. You’ll want to add the Gemfile and Gemfile.lock into your version control, so that anyone else that’s developing with you can also get the same gem versions.

To use the gems in your bundle, just use these two lines:

require "rubygems"
require "bundler/setup"

From there, whenever you require a gem, it’ll be the version from the bundle. If you want Bundler to automatically require all of your gems for you, just ‘Bundler.require‘ and it’ll require the default group of gems.

Rails 3 automatically comes with a Gemfile and bundler support right out of the box. If you want to use Bundler with Rails 2.3, check out the Bundler site for setup instructions.

The combination of gemsets and Bundler will make sure that you don’t have any nasty gem conflicts. Gemsets keep your projects isolated from each other, and Bundler keeps your gems’ versions from interfering with each other. The two work really well together.

I can’t remember which tool I used!

All of these rubies and gemsets can get confusing. Luckily, rvm has an awesome feature to take care of this, too: .rvmrc files. If you put a file named ‘.rvmrc’ in your project’s root directory, when you enter the project, it’ll switch your Ruby version (and gemset) automatically. It’s really easy to use, too. Just put the command you’d use to switch in the file. For example, in the Hackety Hack website project, I have the following .rvmrc:

rvm 1.8.7@hackety-hack.com

Astute readers will notice that I left off the ‘use,’ rvm defaults to ‘use’ if you don’t give it a different command. Check it out:

$ ruby -v
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.4.0]
$ cd hackety-hack.com
$ ruby -v
ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10.4.0]

Super cool. Now you’ll never forget which Ruby you were using, and you don’t even need to switch manually. This is one of the first things that I do when I start a new project in Ruby: Pick a Ruby version, make a gemset with the same name as the project, and set up an .rvmrc. It’s saved me hours of time and headaches.

Multiple projects: super simple

rvm is a fantastic tool to help solve your multiple-ruby woes. It really does make using multiple kinds of Ruby really, really easy. And Bundler makes sure that your gems play nice togther. It’s a great time to be a Rubyist.

I hope you found this article valuable. Feel free to ask questions and give feedback in the comments section of this post. Thanks!

Technorati Tags: , , ,

Posted by Steve Klabnik

Follow me on Twitter to communicate and stay connected

{ 28 comments… read them below or add one }

Nemo157 December 20, 2010 at 1:02 pm

Astute readers will notice that I left off the ‘use,’ rvm defaults to ‘use’ if you don’t give it a different command. Check it out:

However if you put in use you get more info when switching into the folder:

? cd sources/bedrock
info: Using /home/nemo157/.rvm/gems/ruby-1.9.2-p0 with gemset bedrock

Reply

Steve Klabnik December 20, 2010 at 6:39 pm

A good point! Thanks for the detail.

Reply

James Schorr December 21, 2010 at 2:57 am

Thanks for the great RVM tips! I’ll have to get back to using it.

Reply

thecatwasnot December 22, 2010 at 7:20 pm

RVM + Bundler = Ruby Sanity

At first it seems like using more tools would make things more difficult but I don’t think I ‘d know what to do without these tools anymore. Nice writeup, I do things pretty much just like this.

RVM even saved me when I wanted to run a little command line tool that used Chronic (a gem that’s not been updated for ruby 1.9.*) check out rvm’s exec command.

Reply

Chris Kimpton December 24, 2010 at 3:49 pm

Thanks for the great into – would be good to know how you use the gemsets, do you have one per project? And how you get started, eg rvm uses the “global” gemset by default which visible to any gemset’s you create within it – is there a way to move the global gemset into a specific named one and then move/copy specific gems into your new project specific gemsets…

Thanks, Chris rvm-noob.

Reply

Steve Klabnik December 24, 2010 at 8:50 pm

Hey Chris-

Yes, I use one per project, with the project’s name.

I get started by doing exactly what I said in the article. rvm will actually just ‘do the right thing’ for the most part. I don’t think the global gemset is visible to ‘sub gemsets’, though; if you ‘rvm gemset create something’ and then ‘gem list’, you get zero gems.

You can move gemsets around and duplicate them, erase them… check out http://rvm.beginrescueend.com/gemsets/ for all of the fine details.

Reply

Chris Kimpton December 25, 2010 at 8:19 pm

Hi Steve,

Thanks for the reply, though I think the global ones do “follow” you around:

Chris-Kimptons-MacBook-Air:~ kimptoc$ rvm gemset use global
Now using gemset ‘global’
Chris-Kimptons-MacBook-Air:~ kimptoc$ gem list

*** LOCAL GEMS ***

abstract (1.0.0)

zip (2.0.2)
Chris-Kimptons-MacBook-Air:~ kimptoc$ rvm gemset create test
‘test’ gemset created (/Users/kimptoc/.rvm/gems/ruby-1.8.7-p302@test).
Chris-Kimptons-MacBook-Air:~ kimptoc$ rvm gemset use test
Now using gemset ‘test’
Chris-Kimptons-MacBook-Air:~ kimptoc$ gem list

*** LOCAL GEMS ***

abstract (1.0.0)

zip (2.0.2)

I have not installed anymore gems, but the “test” gemset has gems in it.

Seems my mistake has been not using gemsets – so now its all in global :(

~chris

Reply

Steve Klabnik December 26, 2010 at 6:06 am

Hm, strange. The documentation seems to support what you’re saying, I wonder why I’m different?

Reply

Chris Kimpton December 26, 2010 at 2:01 pm

Hi

Given the new ruby versions yesterday I add .rvmrc files to my projects and it didnt work as I thought either – I thought I installed bundler in global but wasnt visible in each project :( . Oh well I now have per project gem configs – onwards to debug my rails2/authlogic/openid glitch :)

~Chris

Aleksey Gureiev January 12, 2011 at 11:46 am

Steve, I have 1.9.2 installed with a bunch of gems in global, and after starting a fresh gemset I can see only one gem — rake 0.8.7 — in there.

Weird — not empty, and not everything either. I maybe missing something.

Nemo157 February 18, 2011 at 8:38 am

I think I figured this out the other day by accident. There is both a “global” gemset and a “default”/blank gemset.

If you `rvm use 1.9.2@global` then install gems they’re available everywhere, however if you `rvm use 1.9.2` then install gems they’re only available when you use the blank gemset.

To see why look at GEM_HOME in `rvm info` after running either. This is the directory to which gems will be installed. Also looking at GEM_PATH you can see that the “global” gemset is always added so that the gems are always available.

Dave December 29, 2010 at 7:18 pm

why do `gem list | grep net-ssh` when you can `gem list net-ssh`?

Reply

Steve Klabnik December 30, 2010 at 9:27 am

My brain tends to think in discrete steps. “Show me my gems, only grab the output that looks like this.” I’m the kinda guy that runs `cat somefile | less` rather than `less somefile` too…

Reply

Dave January 1, 2011 at 12:11 am

Hah I actually do that with `less` too. The mind of a programmer is a strange thing :)

Reply

Les February 15, 2011 at 8:19 pm

Sorry to be dense here, Steve, but when you say “To use the gems in your bundle, just use these two lines”. WHERE exactly does one “use these two lines”? If you could help a bonehead like me understand which file to put these in, it would be much appreciated. Thanks for the article.

Reply

Steve Klabnik February 18, 2011 at 6:59 pm

No worries, I was unclear.

You put those two lines in your script, before you require any of your gems. Probably in the same place you would have used ‘require rubygems’ before.

Reply

Chris Kimpton February 18, 2011 at 4:01 pm

Have you seen this article:

http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-applies/

Which proposes not using gemsets (just use rvm for selecting a specific Ruby) and then vendor the gems within the app using Bundler to provider the per-project “scoping” of gems…

Reply

Steve Klabnik February 18, 2011 at 7:03 pm

I have. I disagree, especially around the use of ‘bundle package.’ Explaining why would take a bit longer than a comment though…

Basically, both of our workflows require doing everything the way we say, though. You can’t half do it one way or the other. His workflow is still _valid_, I just think that my way is better.

Reply

Chris Kimpton February 18, 2011 at 8:03 pm

Cool – thanks for the reply.

My biggest pain is the fresh download for each new gemset – I frequently work while travelling/offline and I need to make sure I prep any gemsets before travelling – perhaps a way to have a local/central cache of gems would be nice, so installs use that first…

Reply

David Merino February 25, 2011 at 3:22 pm

@Chris Kimpton (December 25)
Hi everybody,
I had the same problem Chris had. After create an empty gemset, I got a lot of gems from a global environment. To solve this, I find that there is a file called “.gemrc” in my home directory. This files sets the GEM_HOME and GEM_PATH variables in a powerful level, so, when you execute “gem list –local”, instead of looking for in the environment variables set by “rvm” command, “gem” uses first the variables described in this “.gemrc” file.

I’ve renamed this file by “.gemrcold” and everything works as the fantastic Steve’s post sais. Furthermore, I’ve get create a new gemset with his own rails3 gem installed on it. It’s fantastic!!

I’m not english speaker, so, if somebody has got any problem with this explanation, please, mail-me and I’ll try do better.

Have a nice day!

Reply

Jordan Arseno May 20, 2011 at 1:07 pm

Thanks Steve, excellent article. :D

Reply

Vee July 27, 2011 at 12:07 am

Did all you talk about in the post. Rails seems to have been trashed in the process.
Ge this message when running rails -v. Any idea?
/usr/local/lib/site_ruby/1.8/rubygems/dependency.rb:247:in `to_specs’: Could not find rails (>= 0) amongst [bundler-1.0.15, daemon_controller-0.2.6, fastthread-1.0.7, hashie-1.0.0, linkedin-0.3.1, multi_json-1.0.3, oauth-0.4.5, passenger-3.0.7, rack-1.3.2, rake-0.9.2] (Gem::LoadError)
from /usr/local/lib/site_ruby/1.8/rubygems/dependency.rb:256:in `to_spec’
from /usr/local/lib/site_ruby/1.8/rubygems.rb:1182:in `gem’
from /usr/bin/rails:18

Reply

Steve Klabnik July 27, 2011 at 1:01 am

Hm. Well, it says you can’t find rails. Are you using Rails 2, or Rails 3? If you’re on Rails 2, you need to do this: http://gembundler.com/rails23.html

Reply

Vee July 27, 2011 at 1:11 am

My rails version is 3.0.9. The strange thing is that ruby -v says that I have 1.9.2. But why is it looking at /1.8. The 1.8 version was pre-installed using the apt get way.

Reply

Steve Klabnik July 27, 2011 at 1:14 am

Yeah, I’d say that’s the root of your problem. :/

Reply

Vee July 27, 2011 at 1:26 am

Steve – If you have an old copy installed. You have to remove it using package remover (aptitude or otherwise) completely. Otherwise gem install rails puts the current stuff back in the old version directories.
So,
- I whacked 1.8
- “gem install rails” again
- rails -v asked for sqllite so I ran “bundle install” again.
- Now, ruby -v is 1.9.2; rails -v 3.0.9 and rails server starts.

trans August 12, 2011 at 4:55 pm

Does it ever occur to anyone that the more we isolate gems the more liberty people have for doing “bad things”…. Then one day Joe decides he needs library X and library Y in app A, but lo they don’t play nice b/c they were tested in utter isolation from the entire world and so their authors never realized how badly their code was written.

Ideally we would test with **ALL** gems installed and just keep a blacklist for bad players.

Isolation only makes good sense for deployments where uptime is paramount.

Reply

Aleksey Gureiev August 13, 2011 at 3:36 am

You have a point, but it’s all good on paper only. All gem versions can’t be compatible. Many of those compatible with Rails 2, won’t work with Rails 3 etc, and so you cherry-pick all the time. Having them all in one bucket is asking for trouble. There are cases when gem A implicitly requires gem B and the loader will attempt to pick the most recent version out of installed (rightfully), but in many cases this won’t work (“responders” springs to mind).

One way is to declare and fix the version of gem B int your Gemfile, and the other is to sandbox into a gemset. By explicitly mentioning all of implicit dependencies you loose flexibility and get yourself into a maintenance headache — with the upgrade of gem A you now need to go through all your stuff and review versions of dependencies (gem B) every time.

Insanity, if you think about it.

Reply

Leave a Comment

{ 49 trackbacks }

Previous post:

Next post: