To figure this out, I wanted to know roughly how many pages along with other files were on the site. I knew that it wasn’t going to absolutely massive, so I started by grabbing all of the site’s public content with wget (the details coming from http://linuxreviews.org/quicktips/wget/):

wget -p -r --wait=20 --limit-rate=20K -U Mozilla http://the_site/

I let that run in the background while I did other work.

When it finished I wrote up a little (ugly, unDRY, but took < 5 minutes) Ruby to give me a report by file type and called it file_report.rb based on a skeleton grabbed from http://blogs.sourceallies.com/2009/12/word-counts-example-in-ruby-and-scala/:

require 'yaml'

# Change rootDir to the location of downloaded site
rootDir = "/path/to/roodDir/for/entire/downloaded/site"
raise rootDir + " does not exist" unless File.directory? rootDir

# recursively add files and directories to report_hash based on their type
def files(rootDir, report_hash)
  report_hash['directories'] = report_hash['directories'] || Array.new

  Dir.foreach(rootDir) do |dir|
    if dir != "." && dir != ".."
      dir_path = rootDir + "/" + dir

      if File.directory?(dir_path)
        puts "Processing " + dir

        report_hash['directories'] = report_hash['directories'] << dir_path

        Dir.foreach(dir_path) do |file|
          if file != "." && file != ".."
            file_path = rootDir + "/" + dir + "/" + file
            if File.directory?(file_path)
              report_hash['directories'] = report_hash['directories'] << file_path

              files(file_path, report_hash)
            else
              # add path to report_hash's entry for the file extension
              extension = File.extname(file).sub('.', '')

              report_hash[extension] = report_hash[extension] || Array.new

              report_hash[extension] = report_hash[extension] << file_path
            end
          end
        end
      else
        # add path to report_hash's entry for the file extension
        extension = File.extname(dir).sub('.', '')

        report_hash[extension] = report_hash[extension] || Array.new

        report_hash[extension] = report_hash[extension] << dir_path
      end
    end
  end
end

t1 = Time.now
report_hash = Hash.new
files(rootDir, report_hash)

puts "File type counts: "

report_hash.each do |k, v|
  puts "#{k} : #{v.size}"
end

puts "Writing complete report"
File.open('report.yml', "w") do |f|
    f.write(report_hash.to_yaml)
end

t2 = Time.now
puts "Finished in " + (t2 - t1).to_s + " seconds"

Finally, I called it with:

ruby file_report.rb

Nothing flash, but handy to have when you need it. I saved myself a lot of clicking around on their website.

You need to tell the generator to take a one second snooze, so that the next_migration_string method returns a timestamp that is one second later.

You would think a simple call to the sleep method would do the trick, but because the generator manifest’s record has a special syntax (that relies on method_missing to define recorded actions), you need to do a small tweek by calling the sleep method on the block’s object, i.e. “m.sleep(1)”.

Here’s what it looks like in practice:

class TrolliedMigrationsGenerator < Rails::Generator::Base
  def manifest
    record do |m|
      m.migration_template 'trolleys_migration.rb', 'db/migrate', { :migration_file_name => "create_trolleys" }
      m.sleep(1)
      m.migration_template 'purchase_orders_migration.rb', 'db/migrate', { :migration_file_name => "create_purchase_orders" }
      m.sleep(1)
      m.migration_template 'line_items_migration.rb', 'db/migrate', { :migration_file_name => "create_line_items" }
    end
  end
end

We’ve been using mongo_translatable for awhile as a part of a feature in Kete (an open source Rails app, http://kete.net.nz) for a content translation add-on for awhile now. Thought it was about time to share.

mongo_translatable is a Rails specific I18n model localization mechanism meant to tie-in to existing ActiveRecord models, ala Globalize2, backed by MongoDB rather than an RDBMS.

The gem has only been developed with Rails 2.3.5 up to this point, as that is what my needs are right now, but it would be great if others contributed compatibility with later versions of Rails.

Project information can be found on github:

https://github.com/kete/mongo_translatable

Thanks to Te Reo o Taranaki, the Chinese Association of New Zealand Auckland Branch, and Auckland City Libraries for funding the work.

oembed_provider is a Rails engine to answer oEmbed requests for application media asset models. In other words, this gem allows your application, after configuring the gem and the relevant models, to act as an oEmbed Provider by providing a controller that returns JSON or XML for a given oEmbed consumer request for the specified media asset. This gem does not offer oEmbed consumer functionality.

The gem has only been developed with Rails 2.3.5 up to this point, as that is what my needs are right now, but it would be great if others contributed compatibility with later versions of Rails.

More information is available at https://github.com/kete/oembed_provider.

Issues can be reported at http://kete.lighthouseapp.com/projects/69994-oembed_provider.

This gem was developed for the Kete open source application (http://kete.net.nz) and was funded by pledge campaign to improve media selection from with the rich text editor (i.e. the TinyMCE plugin, look for TinyMCE media selector plugin soon). Horowhenua Library Trust, Wellington City Libraries, Te Reo o Taranaki, Environmental Earth Sciences, CALYX, and many individual contributors. Thanks to all contributors.

This is when Sharl comes in handy.

It is a quick way to list the contents of a dir by size of file with the biggest ones being the last on the list just right there above the command prompt allowing you to easy banish those not needed tar, sql, zip files that are not needed with flourish to free up as much space as one can with the least number of keystrokes

$ cd test # from your rails app root
$ ruby unit/a_model_test.rb

As compared to:

$ ruby -I"lib:test" "/usr/local/Cellar/ruby-enterprise-edition/1.8.7-20090928/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/unit/a_model_test.rb"

Definitely a hand to forehead moment when I read that! Found it in a comment here:

https://rails.lighthouseapp.com/projects/16213/tickets/8

*Assumes your ruby command is set up correctly in your shell’s environment, of course.

You do all your work in a feature branch, then pull it into the master branch. There are numerous topics on the web for this.

But there is a problem with this approach. When you run `git pull origin my_branch`, you end up creating merge commits. Some get around this with `git rebase`, but then you end up rewriting commit shas. So how do you get a clean history without messing up the master branch?

Use `–squash`. Here is a simplified example:

git checkout -b my_branch
touch File1
git commit -a -m "File 1"
touch File2
git commit -a -m "File 2"
git push origin my_branch
git checkout master
git pull --squash origin my_branch
git commit -a -m "File 1 and 2"

Note the last two steps. We use –squash to compact the feature branch commits. This leaves all changes in the other branch in an uncommitted state (which you can then do one final review on). We then go ahead and make a commit.

The result? One commit in master, containing all code from the feature branch.

Now, the caveat: If you have multiple developers who made commits on this branch, you probably want to leave the commits apart for credits sake. But if you’re the only one who’s committed, and you don’t mind squashing the history, this works great!

If you prefer to store your MongoDB data all under your home directory, you might find Mislav’s gist suits your needs instead:

http://gist.github.com/265272

If you prefer installing from source, check out this post:

http://shiftcommathree.com/articles/how-to-install-mongodb-on-os-x

Enjoy.

If you don’t yet use New Relic RPM, you can get a Lite account for free by going to newrelic.com, where you can also test drive New Relic RPM on a real application.

Custom request parameters

New Relic does it’s best to record as much information about the current request as possible. The URL, split down into controller, action, id etc, the request referrer, GET and POST data…. but there is data that it doesn’t know about. Things like the current locale, the user who is currently logged in, or custom session params. Thankfully, there is a solution.

New Relic RPM contains a method for adding custom data to each transaction or error that is sent to it’s servers. Utilize this method inside of a before_filter in your application controller, and you’ll be gathering that extra data in no time. Here is the code we use.

# Collect additional debug details for New Relic RPM is available
# Do this after all other before filters so details are present
before_filter :set_new_relic_custom_parameters
def set_new_relic_custom_parameters
  return unless defined?(NewRelic)
  NewRelic::Agent.add_custom_parameters(
    :locale => (I18n.locale if I18n.locale),
    :account => (logged_in? ? current_user.login : 'guest'),
    :return_to => (session[:return_to] if session[:return_to])
  )
end
private :set_new_relic_custom_parameters

Now, whenever RPM sends information about slow web requests or error reports, you’ll have more data to help replicate the issue.

Passenger Queue Wait

Sometimes the slowness might not be your app, but your app server. How do you tell if it’s serving requests fast enough? New Relic RPM now has the ability to report how long your visits were waiting in the ‘queue’, that is, the time between requesting the site and Apache actually processing that request. Getting this working is fairly simple.

RequestHeader set X-REQUEST_START “%t”

Simply set that in either your main apache.conf file, the VirtualHost directive, or a Directory directive.

Also, ensure that the headers Apache module is enabled. On Debian, you can do this by running the following as root.

# a2enmod headers

That’s all it takes. Now, when RPM sends along information, it’ll send how long each request stayed in the queue, and display it at New Relic RPM’s web interface, on all graphs as ‘Queue Wait’. This should give you an indication of whether your application server is performing well.

Important Note: If you run your application across multiple machines, each machine needs this header, and the clock on each machine must be synced with NTP or similar, or you’ll end up with very incorrect queue times. Users who run the entire site off one box need not worry about this issue.

Garbage Collection delays

I’m not going to go into much detail about Ruby Garbage collection here. Google ‘Ruby Garbage Collection’ if you want to know more. Put simply though, in Ruby, while Garbage Collection runs, it stops the processing of your app. This could lead to slow page requests if it garbage collector, or GC, runs multiple times in the same request.

There are ways to tweak this using settings, but how do you know if it’s a problem to begin with? If you use Ruby Enterprise Edition, or another version of Ruby with the Railsbench GC patches compiled in, then all you need is the following line in your application.

GC.enable_stats if defined?(GC) && GC.respond_to?(:enable_stats)

For Ruby on Rails users, you can stick that at the bottom of your config/environment.rb file. Basically, if GC is defined, and responds to the correct method, then use it. RPM will send this information back to New Relic, where you can view how long GC took in each of your requests, and deal with it accordingly.

Conclusion

Well, that’s all for now. Please feel free to share any additional features which help make New Relic even more useful in the comments.