If you don’t yet use New Relic RPM, you can get a Lite account for free by going to newrelic.com, where you can also test drive New Relic RPM on a real application.

Custom request parameters

New Relic does it’s best to record as much information about the current request as possible. The URL, split down into controller, action, id etc, the request referrer, GET and POST data…. but there is data that it doesn’t know about. Things like the current locale, the user who is currently logged in, or custom session params. Thankfully, there is a solution.

New Relic RPM contains a method for adding custom data to each transaction or error that is sent to it’s servers. Utilize this method inside of a before_filter in your application controller, and you’ll be gathering that extra data in no time. Here is the code we use.

# Collect additional debug details for New Relic RPM is available
# Do this after all other before filters so details are present
before_filter :set_new_relic_custom_parameters
def set_new_relic_custom_parameters
  return unless defined?(NewRelic)
  NewRelic::Agent.add_custom_parameters(
    :locale => (I18n.locale if I18n.locale),
    :account => (logged_in? ? current_user.login : 'guest'),
    :return_to => (session[:return_to] if session[:return_to])
  )
end
private :set_new_relic_custom_parameters

Now, whenever RPM sends information about slow web requests or error reports, you’ll have more data to help replicate the issue.

Passenger Queue Wait

Sometimes the slowness might not be your app, but your app server. How do you tell if it’s serving requests fast enough? New Relic RPM now has the ability to report how long your visits were waiting in the ‘queue’, that is, the time between requesting the site and Apache actually processing that request. Getting this working is fairly simple.

RequestHeader set X-REQUEST_START “%t”

Simply set that in either your main apache.conf file, the VirtualHost directive, or a Directory directive.

Also, ensure that the headers Apache module is enabled. On Debian, you can do this by running the following as root.

# a2enmod headers

That’s all it takes. Now, when RPM sends along information, it’ll send how long each request stayed in the queue, and display it at New Relic RPM’s web interface, on all graphs as ‘Queue Wait’. This should give you an indication of whether your application server is performing well.

Important Note: If you run your application across multiple machines, each machine needs this header, and the clock on each machine must be synced with NTP or similar, or you’ll end up with very incorrect queue times. Users who run the entire site off one box need not worry about this issue.

Garbage Collection delays

I’m not going to go into much detail about Ruby Garbage collection here. Google ‘Ruby Garbage Collection’ if you want to know more. Put simply though, in Ruby, while Garbage Collection runs, it stops the processing of your app. This could lead to slow page requests if it garbage collector, or GC, runs multiple times in the same request.

There are ways to tweak this using settings, but how do you know if it’s a problem to begin with? If you use Ruby Enterprise Edition, or another version of Ruby with the Railsbench GC patches compiled in, then all you need is the following line in your application.

GC.enable_stats if defined?(GC) && GC.respond_to?(:enable_stats)

For Ruby on Rails users, you can stick that at the bottom of your config/environment.rb file. Basically, if GC is defined, and responds to the correct method, then use it. RPM will send this information back to New Relic, where you can view how long GC took in each of your requests, and deal with it accordingly.

Conclusion

Well, that’s all for now. Please feel free to share any additional features which help make New Relic even more useful in the comments.

But Github was building gems for some time, and due to it’s continuing popularity, many well known Ruby on Rails developers and companies switched permanently to Github for their gem building/hosting at the time, so it’s likely that quite a few gems you’ve got installed are from Github.

To help transition over from Github to Gemcutter, Maxim Chernyak wrote a great utility called off_github, which looks at your list of gems, and tells you which ones you’re installed from Github, and whether they can be reinstalled from Gemcutter. It saves a lot of time and effort  than having to do it manually. So here’s how to get started….

Firstly, lets get the latest version of Rubygems. This may not be needed for the off_github gem to function, but it pays to keep this up to date to prevent any issues with newer gems. Usually, you’ll want Rubygems version 1.3.0 or higher. At the time of this post, the current version is 1.3.5.

$ gem –version
1.2.1
$ [sudo] gem update –system
……
$ gem –version
1.3.5

Once that’s done, install the off_github gem. To do that, add Gemcutter as a source and run the gem install command.

$ gem sources -a http://gemcutter.org
$ [sudo] gem install off_github

Note: Rubyforge gems now points to Gemcutter, so the gem source adding isn’t technically needed to install off_github. That said, the off_github gem still checks for the Gemcutter source (it’s a bit outdated) and won’t work without it, so to get around it, install it before running, and you can remove it afterwards.

Now go ahead and run the off_github command. If you don’t use sudo to install gems, then be sure to add –no-sudo the end.

$ off_github [--no-sudo]

This’ll go ahead and return a list of gems that were installed from Github, what there Gemcutter gem name is, and the action it’ll take on that gem (usually reinstall). It’ll then ask you if you want to install. Type ‘Y’, hit enter, and then choose y/n/a at each prompt.

If everything went smoothly, you’re all done. You can now go ahead and delete both the Github and Gemcutter sources.

$ gem sources -r http://gems.github.com
$ gem sources -r http://gemcutter.org

So know, you should have Rubyforge as your only gem source.

$ gem sources
*** CURRENT SOURCES ***
http://gems.rubyforge.org

Additional Material:

• http://www.rubypulse.com/episode-0.16_off_github.html
• http://www.rubypulse.com/episode-0.16.1_off_github_revisited.html

For more details, see this blog post, detailing the symptoms.

http://blog.patrick-morgan.net/2008/09/problems-with-ie7-sessions-not-saved-in.html

The following change removes the activation link from the email issued to the user and sends them an email notification that their application has been received.
From the root of your Joomla site, go to /components/com_user/controller.php (approx line 367) and add:

We also need to change…

to:

The next bit of code emails the account activation link and details through to the site administrator, who gets to decide if the application is successful.
From the root of your Joomla site, go to /components/com_user/controller.php (approx line 524) and change to:

We then need to pass both user and admin notifications through to the script that generates and sends emails.
From the root of your Joomla site, go to /components/com_user/controller.php (approx line 548) Added:

Joomla now issues 2 emails when the signup form is submitted, rather than the usual one.

Some final changes are needed to the language files on the site, so that the email notifications make sense. Make the following changes. From the root of your Joomla site, go to

/language/en-GB/en-GB.com_user.ini (approx line 84) and change:

/language/en-GB/en-GB.com_user.ini (approx line 85) and change:

/language/en-GB/en-GB.com_user.ini (approx line 92) and change:

/language/en-GB/en-GB.com_user.ini (approx line 120) and change:

Finally, the User() helper function needs to be updated so that the activation confirmation email is sent to the user in question rather than the user currently logged into joomla (eg site administrator): To do that, go to:

The hack makes it easy to add as many access keys as you need, and you can offer more than just the basic keystroke options set on most government websites.

Experience with Cucumber

Cucumber has been working out really well, and was very easy to get started with.

sudo gem install cucumber
cd ~/Work/app
script/generate cucumber

That’ll install a new environment with the config.gem lines needed. It’ll also setup a features directory. Open that up, and start writing specs for your app. Simple!

Once I had all the basic step definitions for logging in, logging out, and various paths setup for login page, homepage, etc, writing new features was fairly easy, and executing them is just as easy (rake cucumber). Take for example, the following feature relating to session management.

Scenario: Login with correct details
  When I go to login
  And I fill in "Username" with "jane"
  And I fill in "Password" with "test"
  And I press "Login"
  Then I should see "Successfully logged in."

Very straightforward and easy to read. And as such, the natural language of it means clients and other developers are able to see exactly what has been implemented, and that it is working as expected.

It’s running the entire test suite is fairly fast too (granted, it’s still quite small). I’m testing the core functionality of the new application so far, and it only takes ~40s to run all the features when using ‘progress’ output (the  F..F..E  style output that test/unit does). The more verbose ‘pretty’ output is nice, but quickly fills the terminal, so very easy to miss errors as they happen until the end, and also adds processing time (10s cut off when I switched to progress mode).

At one point in development, I had to test outgoing emails. Thankfully, Cucumber has a really great addition, called email-spec by Ben Mabey. Writing a Cucumber feature for these emails was painless. Here is a snippet from one of the password reset features.

Scenario: Request Reset with correct email
  When I request a password reset for "example@example.com"
  Then I should receive an email
  When I open the email
  And I follow "Continue to reset your password" in the email
  Then I should see "Change My Password"

I’ve made a few contributions to the email-spec project, which have been included in the latest release, enabled email testing for people other than yourself  ( When they open the email… Then they should see …  etc). Makes writing tests even easier.

Other Thoughts

I did find that you duplicate a lot when it comes to adding paths because Cucumber is framework independent. For example, if you have a ‘map.resources :events‘ Cucumber won’t provide you with paths like ‘the events index’ or ‘add a new event’ automatically. Would make a handy Cucumber addition though. In addition, because Cucumber uses Webrat, which has no idea how to interact with Javascript, you still need to fall back to the slow Selenium testing if you’re application relies on Javascript.

Conclusion

Cucumber has been really great and I’m happy to keep using it. If you’ve used Cucumber before, or are going to try it, or perhaps your didn’t like it and went with something else, let me know in the comments. I’m always happy to hear other peoples experiences.

1. My website is a dog’s breakfast, my boss wants it fixed, what do I need to do/think about mys elf before I contact a company?
2. What are the most common mistakes you see being made on organisation’s website?
3. How do you measure value for money on your website?

1. My website is a dog’s breakfast, my boss wants it fixed, what do I need to do/think about myself before I contract a company?

It’s important to realise that a web development company won’t know as much about your organisation and website as you do.

• You need to be the expert on your website.
• To get the result you want, you will need to fill them in with all the information they will need.
  • This is the brief for the project.
• A good brief will stop wasted effort and money from people going down the wrong track and ensure you get the result you need.

High level questions you should be able to answer are about how you define your website. These include:

• What is the purpose of the website?
  • Why have we got a website in the first place?
• What do we hope to develop with our internet presence? For example,
  • Customer or client education
  • Internal communication
  • Information gathering
  • Information dissemination
  • Sales
• Who is our target audience?
  • What do we want them to get from the website?
  • What do we want to get from them?
• What image do we want to portray on the Internet?
  • Do we have corporate branding standards which need to be complied with?
• Who owns the website?
  • Which person in the organisation is responsible for ensuring the website is up to date and loved.

Then on to fixing the dog’s breakfast.

• What is the scope of the project?
  • What exactly does your boss mean by fixing the dog’s breakfast?
  • Does the boss want a total website redevelopment along with a new design or some changes made to the existing site?
• What are the goals of this project?
  • e.g., to fix the dog’ s breakfast so we get less complaints about the site and more registrations for courses.
• What kind of budget have you got?
• What outcome will make this project successful?
  • This should be the goal for the project
• How will we measure success?
  • More sales
  • more email inquiries
  • more site visits
  • less complaints

You should be able to determine the measures of success against the website’s broad purpose and the goals for this project.

Armed with this information you will be able to give a brief to the contractor which lead to an excellent outcome.

2. What are the most common mistakes you see being made on organisation’s website?

Out of date content.

• This is often a symptom of no company ownership of the site.
• Because no-one owns it, no-one has the responsibility of keeping it up to date.

Broken links

It’s hard to find information on the site

• Poor navigation
  • Links aren’t obvious
  • I include flash menus and some dropdown navigation systems in this.
• Poor search engine results
  • page titles, meta information
• Hard to read content
  • Overly wordy and long pages.
    • People don’t read, they scan.
    • So 7 plus or minus 2 words for sentences, get to the point quickly.
      • Use bullets
      • Use white space
  • Headings should summarise the page

Slow loading pages.

• pointless animations
• flash

3. How do you measure value for money on your website?

How do I measure if my website project gave me value for money?

This is the ROI or “Return on investment”. It should be directly related to how you have defined the site’s purpose and objectives.

With the purpose and objectives for the site in mind, you can set up measures of how your website meets it’s goals. For example, if a goal for a website change was to have increased number of applicants for a course, you could measure the number of applications made on-line. Of course, the number of new applicants may be greater than that because some people will be attracted to the course by the information on the website and apply for the course in person, over the phone or by snail mail

Easily measurable items to assess ROI:

• Sales – $
  • If you’re doing online sales
• Leads – for sales or services
  • emails, contacts
• Conversions – how many people visit your site and do something
  • you want people to join, participate in a survey, recommend your site or simply subscribe to your email newsletter?

From your website logs you can get indirect measures such as:

• Usability
  • How usable is your site to your visitors
    • From your website logs you can get information on
      • returning visitors
      • page views per visit
      • time on page
      • time on site
• SEO
  • How much does Google love your site?
    • is it visiting regularly?
    • how many pages is it spidering

And finally:

• Social Media
  • What your users like and what they actually say about you.

Somewhat surprisingly, the task (taking some hardware and netbooting it into a teenage-vandal-proof web kiosk) didn’t appear to be a solved problem – there are still a lot of people rolling their own, I’ve done that once, I didn’t really want to do it again.

Webconverger almost fits the bill, with a couple of caveats:

1. it doesn’t have direct support for netbooting
2. it doesn’t work with the EEEBox b202

As Webc is based on Debian Live (which does support PXE booting), and the b202 works fine with Ubuntu, both these issues seemed like they should be soluble, and after much mucking about, that proved true.

First up, there’s no way that I can see to get Webc to netboot without rebuilding it, so you need to be comfortable with that process.  Be warned that you will consume a fair amount of bandwidth downloading packages (multiple times if you don’t have a local cache), and that the build process is slooooow.

Solving the non functioning X display was relatively straightforward. First, you need to amend your X configuration.  Add a symlink and a file to the includes tree:

inside the xorg.conf you need something like:

then, edit scripts/config and add “noxautoconfig” to the –bootappend-live line, and rebuild.

noxautoconfig disables autogeneration of the X config every time the machine boots – it isn’t ideal, in that it makes your build less portable, but it does give you a working display.  Hopefully, the Intel Xorg driver will be updated soon to make this unnecessary (it appears to be assuming that the b202 has a mobile graphics chipset, therefore it must have an LMDS display).

Netbooting Webc is also comparatively simple – the only major gotcha is that the stock Webc iptables ruleset blocks NFS, which means that your box will hang just before X starts up.   HLT don’t need iptables in their environment,  so I took the path of least resistance and removed the iptables package completely – alternatively, you could amend config/chroot_local-includes/etc/iptables.conf to allow NFS.

So to enable netbooting, you make changes as per the Debian live manual – add something like:

to scripts/config, then edit config/chroot_local-packageslists/webconverger and remove the “iptables” entry.  Finally, if DNS resolution doesn’t work, you may need to remove “nonetworking” from the –bootappend-live line in scripts/config.  Rebuild webc, and be happy.

Other local changes:

Removed iptables, splashy, cups, iceweasel-webconverger and all the wireless drivers from the package list, and added cron, msttcorefonts and iceweasel-webcyourlibrary.  I needed cron because I wanted to be able to power down the kiosks after the library is closed.

I made some further changes to scripts/config file:

the timezone is necessary for the cron jobs to work correctly, noblank turns off the screen blanking, swapon enables any swap partitions on the hard drive, and kioskresetstation restarts Iceweasel after two minutes of idleness.  The -k 686 causes the system to boot the 686 optimised kernel – it’s not critical, the default 486 kernel also works.  All good.

After making changes to the tree, I run

to redo the build.  It’s a slow process, particularly if you use –purge.  Once built, to install I run:

I copy the prompt.cfg because I couldn’t work out how to turn off the pxelinux prompt from within debian-live – this just overwrites the installed file with a simple file that says:

Hardware setup

(mostly these are notes to myself so that when I have to repeat the process in the future I won’t forget steps):

Updated the BIOS (press Alr-F2 as the machine boots to run the updater), then made some changes in the BIOS – enabled LAN BootROM, disabled WLAN, Enabled ACPI 2.0 support, enabled Power on by LAN, Full screen logo disabled, Express gate disabled.

I used Billix to boot Ubuntu, shrink the second NTFS partition to 17.2GB, make an ext3 fs (15GB) and the remainder (1.7GB) into  swap, and installed Ubuntu server.  None of this is crucial, but I wanted grub installed, and wanted some swap space, and ubuntu is as good a way of achieving this as any.  Changes to /boot/grub/menu.lst: timeout 15, hiddenmenu, password (), alternative=false, lockold=true, howmany=1, memtest86=false, then run sudo update-grub.  Then add “title Reboot/nreboot” as the first option, and “lock” to all the remaining options.

I wanted grub because sometimes on warm boot the ethernet interface doesn’t bring up link, and PXE falls through to the hard drive.  I don’t know why this is – it might be a BIOS issue, it might be an incompatibility between my test switch and the b202 – either way, with grub I can force the machine to reboot and try again, rather than falling through to an unconfigured Windows install.

Verify grub is giving the correct behavior (password within 15 seconds to boot Windows or Ubuntu, otherwise reboot), go back to the BIOS, change the Boot device priority, first: Network boot, second HDD, USB/CDROM disabed, and set a Supervisor password on the BIOS so that it can’t be modified.  Reboot

When the PXE config text comes up, press Shift F10 to configure, disable the Config message, and set the Show message time to 1 Sec.  Add the MAC of the machine to DHCP/WOL configs, and test.

You could write up your own logging mechanism, with or without using the Logger gem. However, if what you are working on is run by the Rails app and thus has the apps global constants available, you could simply tie it into the existing logger object in the Rails app like so in your file:

Then you can use logger.debug, logger.info, etc. to your heart’s content. One caveat, if you are working on a gem or a something more general that won’t necessarily always be run in the context of Rails, then you will want to pull your use of logger out before distributing your code. Otherwise you add a dependency on Rails that you may not intend.

The Gist service is a good example. First they added command line support for it, then they simultaneously added in-editor support in Textmate, vim, and my own personal favorite emacs. The blog post announcing gist support in emacs and vim is here.

So how do you add the gist support to emacs?

First, you’ll need an account on github.com and have set up your ~/.gitconfig as outlined in here.

Then download or clone the gist.el file from http://github.com/defunkt/gist.el, copy only the gist.el file from that repository to someplace in your emacs load path (in my case /Users/walter/Library/Preferences/Aquamacs Emacs/ because I use Aquamacs on a Mac), and add a line to your ~/.emacs file or in my case /Users/walter/Library/Preferences/Aquamacs Emacs/Preferences.el that looks like this:

Then you have to either restart your emacs program or do M-x load-library and answer prompt with gist for the new gist commands to be available.

Now you have M-x commands like these:

gist-view-gist

view gists after they’re posted

gist-region

Post the current region as a new paste at gist.github.com
Copies the URL into the kill ring.

gist-region-private

Post the current region as a new private paste at gist.github.com
Copies the URL into the kill ring.

gist-buffer

Post the current buffer as a new paste at gist.github.com.
Copies the URL into the kill ring.

gist-buffer-private

Post the current buffer as a new private paste at gist.github.com.
Copies the URL into the kill ring.

gist-region-or-buffer

Post either the current region, or if mark is not set, the current buffer as a new paste at gist.github.com
Copies the URL into the kill ring.

gist-region-or-buffer-private

you can probably guess…

gist-fetch

Given an gist id, fetches a Gist and inserts it into a new buffer
If the Gist already exists in a buffer, switches to it.

Very useful stuff for collaboration, but without leaving your editor.

Enjoy,
Walter