The ultimate photo backup solution*

* If you use a Mac, iPhoto and Flickr

I decided it was time to start backing up ALL my photos to Flickr, not just the ones that I decided were good enough to take the time to upload. Apple’s Photo Stream is awesome: any photo taken on any of your devices is shared and available among all of them instantly. The only problem is that it only stores the last 1,000 photos. You can get around this limitation if you use iPhoto: at the end of each month it will automatically create an event named after that month: “Aug 2012 Photo Stream” is the first such event I have on my laptop.

So if we could just automate getting those photos to Flickr we’d be all set. Flickr recently went through a huge redesign and upped everyone’s storage to 1TB, even those without a Pro account, so it’s high time to use it as a “backup every photo ever taken” solution. I figured I’d write a little Ruby script to search through the iPhoto library and upload each photo to Flickr. It turns out someone already did it:

https://github.com/jawj/iphoto-flickr

You start the script and it uses Applescript to read the contents of your photo album(s) and starts uploading each photo that isn’t already on Flickr. It never deletes photos and never writes over ones that are already there. Just get yourself an API key and the iphoto-flickr script will walk you through the steps to setup your account and start uploading.

Huge thanks to George MacKerron for writing this script and giving it away to the world!

Testing Controller Concerns in Rails

I can find very little documentation in the world about how to test concerns, specifically controller concerns, in Rails. I’ve written a previous post on testing them, but I’ve since come up with what I think is an even better technique. Concerns are just a module like any other module in Ruby, so in theory they should be as east to test, but there is so much other stuff involved in making a request through a controller that I feel more comfortable relying to on the conventions Rails already has in place for testing that complete stack and confirming that my concern behaves as expected.

This technique involves creating a class in your test to serve as a controller, then dynamically adding actions to it. Those actions then call whatever method in your concern that you’re actually trying to test.

Here’s our concern:

module Concerns
  module SessionManagement
    extend ActiveSupport::Concern

    def login_required
      redirect_to login_url unless session[:user_id].present?
    end
  end
end

To use this in a normal controller we’d have something like:

class AdminController < ActionController::Base
  include Concerns::SessionManagement

before_filter :login_required end

So our concern nicely encapsulates the requirement that a user must be logged in to do anything with the AdminController. But how do we test it? We could write our tests in admin_controller_test.rb, but what if we decide at some point that we don’t need this concern there and instead use it somewhere else? We’d have to move all the tests. Our tests would also become affected by any before_filters we may add to AdminController one day which could have an unexpected effect on whether or not our concern’s methods are called. Too many potential side effects!

My solution was to test the concern in isolation and create a fake controller on the fly to use in my tests.

Here’s what the test controller code looks like:

class SessionManagementTestController < ActionController::Base
  include Concerns::SessionManagement

  def initialize(method_name, &method_body)
    self.class.send(:define_method, method_name, method_body)
  end

end

We create a class named after the concern with “TestController” appended, then include the concern we’re testing (just like in the real controller). The magic lies in the initialize method. Here’s where we’re going to create new controller actions on the fly. It accepts the name of the new action, then a block defining the code that the action will contain.

A usage example should make things clear:

test 'send user to login page if not logged in' do
  @controller = SessionManagementTestController.new(:index) { login_required }
  get :index
  assert_redirected_to login_url
end

Normally @controller is created for us in a functional test, but when testing concerns in this way we need to initialize our own. When we initialize it we pass the name of the action we want to create: “index” in this case. Then in the block we include whatever code we want the action to run: for this test we want to call the `login_required` method of our concern.

After setting up @controller we just write a regular ol’ functional test: `get` the action and assert something happened. Here’s the whole test file (which lives at test/functional/concerns/session_management_test.rb):

require 'test_helper'
  class Concerns::SessionManagementTest < ActionController::TestCase
  
  test 'send user to login page if not logged in' do
    @controller = SessionManagementTestController.new(:index) { login_required }
    get :index
    assert_redirected_to login_url
  end
  
  test 'allow a logged in user to continue about their business' do
    session[:user_id] = 1
    @controller = SessionManagementTestController.new(:index) { login_required; render :nothing => true }
    get :index
    assert_response :success
  end
    
end

class SessionManagementTestController < ActionController::Base
  include Concerns::SessionManagement

  def initialize(method_name, &method_body)
    self.class.send(:define_method, method_name, method_body)
  end

end

Note that the inclusion of our test controller goes after the test class, not inside of it. In the second test the user would not be redirected (since they were considered logged in) so we need to actually have something to render or we’ll get a “Missing Template” error.

And that’s it! This technique has been a great help while writing tests for popexpert. We have plenty of concerns and I can sleep at night knowing that they have adequate test coverage!

Ruby gem for using an SHT15 temp/humidity sensor with Raspberry Pi

Last night I released a Ruby gem for communicating with an SHT15 temperature and humidity sensor from a Raspberry Pi. There were plenty of C and Python libraries out there, but no Ruby. Thanks to Gordon for sharing his C code which I used to figure out the data/clock signal sequence.

The layout of the gem allows for more sensors to be added as time goes by. It’s pretty simple to use:

require 'pi_sensor' 

sensor = PiSensor::SHT15.new :clock => 0, :data => 1 

puts " Temperature: #{sensor.temperature}°C" 
puts " Humidity: #{sensor.humidity}%"

Check out the README over at GitHub for more details:

https://github.com/cannikin/pi_sensor

How to test stuff in ApplicationController (or before_filters, or concerns…)

Update 11/13/13

Several people have commented that using Application.routes.draw broke their other tests. I’ve updated the code below to use Application.routes.append instead.

Update  6/27/12

When I updated my codebase from Rails 3.2.3 to 3.2.6 these tests stopped working. Now you need to manually add a route before Rails will be able to hit your test controller. The code below has been updated.

For a while now I’ve been trying to figure out how to test methods in ApplicationController without going through another controller first. For example, if I have a foo method in ApplicationController I would have to test that through my UserControllerTest—you can’t test ApplicationController directly. I found a neat way to get around this, however. This file would go in test/functionals/app_controller_test.rb (by default the test system will automatically try and instantiate a controller with the name name as the class of the test, so I named this AppControllerTest instead of ApplicationControllerTest):

require 'test_helper'

class TestController < ApplicationController
  def index; end
end

class AppControllerTest < ActionController::TestCase

  MyApp::Application.routes.append do
    controller :test do
      get 'test/index' => :index
    end
  end

  setup do
    @controller = TestController.new
  end

  should "return nil if there is no district subdomain" do
    get :index
    assert_equal 'bar', @controller.send(:foo)
  end

end

(I’m using shoulda here but this will work in plain Test::Unit as well. I can’t speak for RSpec.) In setup we create a fake controller on the fly and give a single index action that does nothing. We add a route to our application so Rails will recognize it as a valid destination. In our setup we set @controller to be a new instance of that fake controller. We make a request to our new index action which will populate @controller.request and @controller.response for us. Then we simply call the method directly on the controller. Using send() lets us also call private methods we may have in our ApplicationController.

Note that any before/after filters you have defined in ApplicationController will still run, but all they’ll really do is maybe add a Location header to @controller.response (like if your before_filter checked that someone was logged in) or something similar: get :index should complete with no problem (probably returning a 404 since there’s no corresponding route) and you’ll be able to manually call methods on the ApplicationController just as if it was a regular ol’ Ruby class.

Using Rails 3 route helpers outside of a controller/view

UPDATE 12/15/13 — As Lenin mentions below, url helpers are automatically available in Rails 4 without any trickery!

I’m working on a job for delayed_job which sends an email including a link back to the site. I wanted to use Rails’s built-in URL helpers like new_post_path and edit_blog_url. After a little poking around I found this method call which gives you access to all the helpers from anywhere:

  Rails.application.routes.url_helpers

Append your helper call to that method chain to get the final string:

  Rails.application.routes.url_helpers.new_post_path 
  => '/posts/new'

  Rails.application.routes.url_helpers.edit_person_url(@person, :host => 'server.com') 
  => 'http://server.com/people/3/edit'

If you use the _url version of the helper you’ll need to provide the host at the same time you include any other parameters that helper might need.

encosion - Ruby Library for the Brightcove API

I’m working on a project at work that allows a user to upload videos. We use Brightcove to host and present our video so I wrote a little library for working with their API. It’s called encosion and it’s over at Github . I haven’t completely duplicated the functionality of theirAPI yet since all I needed to do was read and write video, but I encourage others to contribute and add on to the project!

Gasohol library released - easily search a Google Appliance Server with Ruby

Another release today, this time it’s a small Ruby library for searching a Google Appliance Server called Gasohol . I’ve been working on a prototype search at my job as a full Rails app, but I removed the part that searches and parses results from the GSA and open sourced it.

As always, check out the readme for usage instructions. I plan to turn this into a gem eventually, but for now you’ll need to pull the files down manually and drop into the rest of your code. More to come!

Radiant extension for searching flickr

I’ve just released an extension for Radiant CMS which lets you search flickr and returns an unordered list of the thumbnails that match. The extension is listed in the Radiant Extension Registry and is hosted on its own page on Github . Just create a directory in/vendor/extensions called something like flickr and drop the extension in there. Restart Radiant and you’re good to go!

Check out the README for usage.

Cache anything (easily) with Rails and memcached

Update 6/9/2010 – A similar mechanism was made available in Rails 2.1 using Rails.cache. See Railscast #115 for an introduction.

When I first heard about memcached I was excited because of the promise of a very fast caching mechanism that could store anything, but was a little frightened by the idea of dipping my toes into the caching world. Isn’t caching hard? Not the actual process of storing something. Expiring from cache is a different story. I’m only going to deal with the first problem.

So, how easy is it? First, get memcached. If you’re running something like Ubuntu this is as easy as:

sudo apt-get install memcached

Of if you have Macports on your Mac then:

sudo port install memcached

Once you have memcache you’ll want to start it running:

memcached -vv

The -vv puts memcache in Very Verbose mode so you get to see all the action. You’ll run this as a daemon once you’re ready to go for real (replace -vv with -d).

My example below uses Ruby and Rails but there are memcache libraries for just about every language out there . For Ruby we’re using memcache-client and you’ll need the gem:

sudo gem install memcache-client

Okay, all the hard stuff is out of the way. Rails already tries to require 'memcache' so you don’t need to worry about that at all. At the end of your config/environment.rb file create an instance of memcache and assign it to a constant so it’s around whenever we need it:

CACHE = MemCache.new('127.0.0.1')

Now we’ll add a simple method to our application controller so that this new caching mechanism is available to all of our controllers. Make sure this method is private:

private
def data_cache(key)
  unless output = CACHE.get(key)
    output = yield
    CACHE.set(key, output, 1.hour)
  end
  return output
end

memcache stores everything as simple key/value pairs. You either ask memcache if it has something for a given key, or give it a value along with a key to store. This method will attempt to get the value out of cache and only if it’s not found then will run the block you use when calling it (that’s next) and store the results of that block into cache with the given key and telling it to expire after 1.hour. Every time you ask for that key within the next hour you’ll get the same result from memory. After that memcache will store it again for another hour.

As a very simple example, you could use this in your controllers like so:

result = data_cache('foo') { 'Hello, world!' }

So, if the cache contains a key called ‘foo’ it will return it to result. If not, then it will store Hello, world! with the key foo and also return to result. Either way, result will end up with what you want (the contents of the block). If you take a look at the output of memcache back at the terminal you’ll see it trying to get and store data by the key.

Storing a simple string doesn’t do us much good, so let’s try a real world example. At work I’m working on a new search with a Google GSA. We get some keywords and other search parameters from the user, send them over to the GSA, parse the result, and display to the user. We only update our search index once per day, so if more than one person searches for “running san diego” there’s no reason to go to theGSA each and every time—the result hasn’t changed since it was asked for the earlier in the day. So we cache the result for 24 hours.

A search result on our system can be uniquely identified by the URL that was generated from the user’s search parameters. We use thisURL as the key to memcache. A regular URL can be pretty long so we take the MD5 hash of it and use that as the key:

md5 = Digest::MD5.hexdigest(request.request_uri)
output = data_cache(md5) { SEARCH.search(keywords, options) }

SEARCH is the library that talks to the GSA and parses the result (which I hope to open source soon) (available here). What did this do for our response times? Our GSA box is currently located in Australia (it’s a loaner). Between the network latency of talking to the GSA and receiving and parsing the huge XML file it returns (50kb), most requests were taking 1500 to 2000 milliseconds (not including going through the regular Rails stack to get the page back to the user). With memcache in place the same results come back in 1 millisecond. One. That’s three orders of magnitude difference!

As you can see, adding memcache to your Rails app is stupidly simple and you can start benefiting from it right away. Don’t be scared of caching!

Update I updated the post to use data_cache rather than cache as that’s already the name of the fragment caching method in Rails.