Katipo Developers Blog

Today I had to evaluate whether it was worth it to use Kete’s bulk import facility to migrate an existing site’s content to Kete or whether to just have someone drag the content over page by page.

To figure this out, I wanted to know roughly how many pages along with other files were on the site. I knew that it wasn’t going to absolutely massive, so I started by grabbing all of the site’s public content with wget (the details coming from http://linuxreviews.org/quicktips/wget/):

wget -p -r --wait=20 --limit-rate=20K -U Mozilla http://the_site/

I let that run in the background while I did other work.

When it finished I wrote up a little (ugly, unDRY, but took < 5 minutes) Ruby to give me a report by file type and called it file_report.rb based on a skeleton grabbed from http://blogs.sourceallies.com/2009/12/word-counts-example-in-ruby-and-scala/:

require 'yaml'

# Change rootDir to the location of downloaded site
rootDir = "/path/to/roodDir/for/entire/downloaded/site"
raise rootDir + " does not exist" unless File.directory? rootDir

# recursively add files and directories to report_hash based on their type
def files(rootDir, report_hash)
  report_hash['directories'] = report_hash['directories'] || Array.new

  Dir.foreach(rootDir) do |dir|
    if dir != "." && dir != ".."
      dir_path = rootDir + "/" + dir

      if File.directory?(dir_path)
        puts "Processing " + dir

        report_hash['directories'] = report_hash['directories'] << dir_path

        Dir.foreach(dir_path) do |file|
          if file != "." && file != ".."
            file_path = rootDir + "/" + dir + "/" + file
            if File.directory?(file_path)
              report_hash['directories'] = report_hash['directories'] << file_path

              files(file_path, report_hash)
            else
              # add path to report_hash's entry for the file extension
              extension = File.extname(file).sub('.', '')

              report_hash[extension] = report_hash[extension] || Array.new

              report_hash[extension] = report_hash[extension] << file_path
            end
          end
        end
      else
        # add path to report_hash's entry for the file extension
        extension = File.extname(dir).sub('.', '')

        report_hash[extension] = report_hash[extension] || Array.new

        report_hash[extension] = report_hash[extension] << dir_path
      end
    end
  end
end

t1 = Time.now
report_hash = Hash.new
files(rootDir, report_hash)

puts "File type counts: "

report_hash.each do |k, v|
  puts "#{k} : #{v.size}"
end

puts "Writing complete report"
File.open('report.yml', "w") do |f|
    f.write(report_hash.to_yaml)
end

t2 = Time.now
puts "Finished in " + (t2 - t1).to_s + " seconds"

Finally, I called it with:

ruby file_report.rb

Nothing flash, but handy to have when you need it. I saved myself a lot of clicking around on their website.

If you have created a Rails generator that needs to include more than one migration_template in its record block, I’ve found a trick so you don’t run into a “Multiple migrations have the version number” error when running db:migrate.

You need to tell the generator to take a one second snooze, so that the next_migration_string method returns a timestamp that is one second later.

You would think a simple call to the sleep method would do the trick, but because the generator manifest’s record has a special syntax (that relies on method_missing to define recorded actions), you need to do a small tweek by calling the sleep method on the block’s object, i.e. “m.sleep(1)”.

Here’s what it looks like in practice:

class TrolliedMigrationsGenerator < Rails::Generator::Base
  def manifest
    record do |m|
      m.migration_template 'trolleys_migration.rb', 'db/migrate', { :migration_file_name => "create_trolleys" }
      m.sleep(1)
      m.migration_template 'purchase_orders_migration.rb', 'db/migrate', { :migration_file_name => "create_purchase_orders" }
      m.sleep(1)
      m.migration_template 'line_items_migration.rb', 'db/migrate', { :migration_file_name => "create_line_items" }
    end
  end
end

Version 0.1 of the mongo_translatable gem is out.

We’ve been using mongo_translatable for awhile as a part of a feature in Kete (an open source Rails app, http://kete.net.nz) for a content translation add-on for awhile now. Thought it was about time to share.

mongo_translatable is a Rails specific I18n model localization mechanism meant to tie-in to existing ActiveRecord models, ala Globalize2, backed by MongoDB rather than an RDBMS.

The gem has only been developed with Rails 2.3.5 up to this point, as that is what my needs are right now, but it would be great if others contributed compatibility with later versions of Rails.

Project information can be found on github:

https://github.com/kete/mongo_translatable

Thanks to Te Reo o Taranaki, the Chinese Association of New Zealand Auckland Branch, and Auckland City Libraries for funding the work.

Version 0.1 of the oembed_provider gem is out.

oembed_provider is a Rails engine to answer oEmbed requests for application media asset models. In other words, this gem allows your application, after configuring the gem and the relevant models, to act as an oEmbed Provider by providing a controller that returns JSON or XML for a given oEmbed consumer request for the specified media asset. This gem does not offer oEmbed consumer functionality.

The gem has only been developed with Rails 2.3.5 up to this point, as that is what my needs are right now, but it would be great if others contributed compatibility with later versions of Rails.

More information is available at https://github.com/kete/oembed_provider.

Issues can be reported at http://kete.lighthouseapp.com/projects/69994-oembed_provider.

This gem was developed for the Kete open source application (http://kete.net.nz) and was funded by pledge campaign to improve media selection from with the rich text editor (i.e. the TinyMCE plugin, look for TinyMCE media selector plugin soon). Horowhenua Library Trust, Wellington City Libraries, Te Reo o Taranaki, Environmental Earth Sciences, CALYX, and many individual contributors. Thanks to all contributors.

I’ve been working with Rails and Ruby since 2006 and I’m surprised I hadn’t put this together for myself:

$ cd test # from your rails app root
$ ruby unit/a_model_test.rb

As compared to:

$ ruby -I"lib:test" "/usr/local/Cellar/ruby-enterprise-edition/1.8.7-20090928/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/unit/a_model_test.rb"

Definitely a hand to forehead moment when I read that! Found it in a comment here:

https://rails.lighthouseapp.com/projects/16213/tickets/8

*Assumes your ruby command is set up correctly in your shell’s environment, of course.

I’ve moved from MacPorts to Homebrew which includes a recipe for installing MongoDB. After installing Homebrew, just run this as your normal user:

If you prefer to store your MongoDB data all under your home directory, you might find Mislav’s gist suits your needs instead:

http://gist.github.com/265272

If you prefer installing from source, check out this post:

http://shiftcommathree.com/articles/how-to-install-mongodb-on-os-x

Enjoy.

Here’s a quick one. Say you are debugging a bit of code in a plugin that doesn’t fall under the Rails app you are working on’s ActiveSupport context and thus “logger.debug” is not available to you.

You could write up your own logging mechanism, with or without using the Logger gem. However, if what you are working on is run by the Rails app and thus has the apps global constants available, you could simply tie it into the existing logger object in the Rails app like so in your file:

Then you can use logger.debug, logger.info, etc. to your heart’s content. One caveat, if you are working on a gem or a something more general that won’t necessarily always be run in the context of Rails, then you will want to pull your use of logger out before distributing your code. Otherwise you add a dependency on Rails that you may not intend.

You have to give Logical Awesome credit for how much work they do to integrate GitHub’s services with tools that developers use.

The Gist service is a good example. First they added command line support for it, then they simultaneously added in-editor support in Textmate, vim, and my own personal favorite emacs. The blog post announcing gist support in emacs and vim is here.

So how do you add the gist support to emacs?

First, you’ll need an account on github.com and have set up your ~/.gitconfig as outlined in here.

Then download or clone the gist.el file from http://github.com/defunkt/gist.el, copy only the gist.el file from that repository to someplace in your emacs load path (in my case /Users/walter/Library/Preferences/Aquamacs Emacs/ because I use Aquamacs on a Mac), and add a line to your ~/.emacs file or in my case /Users/walter/Library/Preferences/Aquamacs Emacs/Preferences.el that looks like this:

Then you have to either restart your emacs program or do M-x load-library and answer prompt with gist for the new gist commands to be available.

Now you have M-x commands like these:

gist-view-gist

view gists after they’re posted

gist-region

Post the current region as a new paste at gist.github.com
Copies the URL into the kill ring.

gist-region-private

Post the current region as a new private paste at gist.github.com
Copies the URL into the kill ring.

gist-buffer

Post the current buffer as a new paste at gist.github.com.
Copies the URL into the kill ring.

gist-buffer-private

Post the current buffer as a new private paste at gist.github.com.
Copies the URL into the kill ring.

gist-region-or-buffer

Post either the current region, or if mark is not set, the current buffer as a new paste at gist.github.com
Copies the URL into the kill ring.

gist-region-or-buffer-private

you can probably guess…

gist-fetch

Given an gist id, fetches a Gist and inserts it into a new buffer
If the Gist already exists in a buffer, switches to it.

Very useful stuff for collaboration, but without leaving your editor.

Enjoy,
Walter

A natural inclination when one starts programming in Ruby is to use “and” and “or” instead of the “&&” and “||” to increase readability. However, “and” is not a drop in syntactic synonym with “&&” and the same goes for “or”.

James pointed out a good blog post that explains the issues around order of precedence with these operators:

http://blog.jayfields.com/2007/08/ruby-operator-precedence-of-and-which.html

It’s worth reading the comments.

It should also be noted that the Ruby on Rails source style guide has a preference for “&&” and “||”:
http://rails.lighthouseapp.com/projects/8994/source-style

I just had to resolve my first conflict in git. It’s a little more convoluted than with subversion, but slicker overall.

So you run into a conflict during a merge (implied by this git pull command):

First up, you need to have your git config point at opendiff on Mac OS X. Do the following if you haven’t already (you only have to do this once and it will be set permanently):

Then from the shell run this:

And then you should see output something like this:

Hit return and the OpenDiff application should startup with the conflicted file. It will have side by side comparisons of local version of the file and the remote version of the file with nifty highlighted areas where things are different. Select the first highlighted region, go down to the bottom right and choose the appropriate side in the “Action” menu, then proceed on to the next one, and so on until you have taken care of all differences. Then click File > Save Merge.

Back on the shell you are ready to do commit your resolved file. Do “git commit” and you enter an editor to put in your commit message.

Move on to other tasks, you’re done.