You do all your work in a feature branch, then pull it into the master branch. There are numerous topics on the web for this.

But there is a problem with this approach. When you run `git pull origin my_branch`, you end up creating merge commits. Some get around this with `git rebase`, but then you end up rewriting commit shas. So how do you get a clean history without messing up the master branch?

Use `–squash`. Here is a simplified example:

git checkout -b my_branch
touch File1
git commit -a -m "File 1"
touch File2
git commit -a -m "File 2"
git push origin my_branch
git checkout master
git pull --squash origin my_branch
git commit -a -m "File 1 and 2"

Note the last two steps. We use –squash to compact the feature branch commits. This leaves all changes in the other branch in an uncommitted state (which you can then do one final review on). We then go ahead and make a commit.

The result? One commit in master, containing all code from the feature branch.

Now, the caveat: If you have multiple developers who made commits on this branch, you probably want to leave the commits apart for credits sake. But if you’re the only one who’s committed, and you don’t mind squashing the history, this works great!

If you don’t yet use New Relic RPM, you can get a Lite account for free by going to newrelic.com, where you can also test drive New Relic RPM on a real application.

Custom request parameters

New Relic does it’s best to record as much information about the current request as possible. The URL, split down into controller, action, id etc, the request referrer, GET and POST data…. but there is data that it doesn’t know about. Things like the current locale, the user who is currently logged in, or custom session params. Thankfully, there is a solution.

New Relic RPM contains a method for adding custom data to each transaction or error that is sent to it’s servers. Utilize this method inside of a before_filter in your application controller, and you’ll be gathering that extra data in no time. Here is the code we use.

# Collect additional debug details for New Relic RPM is available
# Do this after all other before filters so details are present
before_filter :set_new_relic_custom_parameters
def set_new_relic_custom_parameters
  return unless defined?(NewRelic)
  NewRelic::Agent.add_custom_parameters(
    :locale => (I18n.locale if I18n.locale),
    :account => (logged_in? ? current_user.login : 'guest'),
    :return_to => (session[:return_to] if session[:return_to])
  )
end
private :set_new_relic_custom_parameters

Now, whenever RPM sends information about slow web requests or error reports, you’ll have more data to help replicate the issue.

Passenger Queue Wait

Sometimes the slowness might not be your app, but your app server. How do you tell if it’s serving requests fast enough? New Relic RPM now has the ability to report how long your visits were waiting in the ‘queue’, that is, the time between requesting the site and Apache actually processing that request. Getting this working is fairly simple.

RequestHeader set X-REQUEST_START “%t”

Simply set that in either your main apache.conf file, the VirtualHost directive, or a Directory directive.

Also, ensure that the headers Apache module is enabled. On Debian, you can do this by running the following as root.

# a2enmod headers

That’s all it takes. Now, when RPM sends along information, it’ll send how long each request stayed in the queue, and display it at New Relic RPM’s web interface, on all graphs as ‘Queue Wait’. This should give you an indication of whether your application server is performing well.

Important Note: If you run your application across multiple machines, each machine needs this header, and the clock on each machine must be synced with NTP or similar, or you’ll end up with very incorrect queue times. Users who run the entire site off one box need not worry about this issue.

Garbage Collection delays

I’m not going to go into much detail about Ruby Garbage collection here. Google ‘Ruby Garbage Collection’ if you want to know more. Put simply though, in Ruby, while Garbage Collection runs, it stops the processing of your app. This could lead to slow page requests if it garbage collector, or GC, runs multiple times in the same request.

There are ways to tweak this using settings, but how do you know if it’s a problem to begin with? If you use Ruby Enterprise Edition, or another version of Ruby with the Railsbench GC patches compiled in, then all you need is the following line in your application.

GC.enable_stats if defined?(GC) && GC.respond_to?(:enable_stats)

For Ruby on Rails users, you can stick that at the bottom of your config/environment.rb file. Basically, if GC is defined, and responds to the correct method, then use it. RPM will send this information back to New Relic, where you can view how long GC took in each of your requests, and deal with it accordingly.

Conclusion

Well, that’s all for now. Please feel free to share any additional features which help make New Relic even more useful in the comments.

But Github was building gems for some time, and due to it’s continuing popularity, many well known Ruby on Rails developers and companies switched permanently to Github for their gem building/hosting at the time, so it’s likely that quite a few gems you’ve got installed are from Github.

To help transition over from Github to Gemcutter, Maxim Chernyak wrote a great utility called off_github, which looks at your list of gems, and tells you which ones you’re installed from Github, and whether they can be reinstalled from Gemcutter. It saves a lot of time and effort  than having to do it manually. So here’s how to get started….

Firstly, lets get the latest version of Rubygems. This may not be needed for the off_github gem to function, but it pays to keep this up to date to prevent any issues with newer gems. Usually, you’ll want Rubygems version 1.3.0 or higher. At the time of this post, the current version is 1.3.5.

$ gem –version
1.2.1
$ [sudo] gem update –system
……
$ gem –version
1.3.5

Once that’s done, install the off_github gem. To do that, add Gemcutter as a source and run the gem install command.

$ gem sources -a http://gemcutter.org
$ [sudo] gem install off_github

Note: Rubyforge gems now points to Gemcutter, so the gem source adding isn’t technically needed to install off_github. That said, the off_github gem still checks for the Gemcutter source (it’s a bit outdated) and won’t work without it, so to get around it, install it before running, and you can remove it afterwards.

Now go ahead and run the off_github command. If you don’t use sudo to install gems, then be sure to add –no-sudo the end.

$ off_github [--no-sudo]

This’ll go ahead and return a list of gems that were installed from Github, what there Gemcutter gem name is, and the action it’ll take on that gem (usually reinstall). It’ll then ask you if you want to install. Type ‘Y’, hit enter, and then choose y/n/a at each prompt.

If everything went smoothly, you’re all done. You can now go ahead and delete both the Github and Gemcutter sources.

$ gem sources -r http://gems.github.com
$ gem sources -r http://gemcutter.org

So know, you should have Rubyforge as your only gem source.

$ gem sources
*** CURRENT SOURCES ***
http://gems.rubyforge.org

Additional Material:

• http://www.rubypulse.com/episode-0.16_off_github.html
• http://www.rubypulse.com/episode-0.16.1_off_github_revisited.html

For more details, see this blog post, detailing the symptoms.

http://blog.patrick-morgan.net/2008/09/problems-with-ie7-sessions-not-saved-in.html

Experience with Cucumber

Cucumber has been working out really well, and was very easy to get started with.

sudo gem install cucumber
cd ~/Work/app
script/generate cucumber

That’ll install a new environment with the config.gem lines needed. It’ll also setup a features directory. Open that up, and start writing specs for your app. Simple!

Once I had all the basic step definitions for logging in, logging out, and various paths setup for login page, homepage, etc, writing new features was fairly easy, and executing them is just as easy (rake cucumber). Take for example, the following feature relating to session management.

Scenario: Login with correct details
  When I go to login
  And I fill in "Username" with "jane"
  And I fill in "Password" with "test"
  And I press "Login"
  Then I should see "Successfully logged in."

Very straightforward and easy to read. And as such, the natural language of it means clients and other developers are able to see exactly what has been implemented, and that it is working as expected.

It’s running the entire test suite is fairly fast too (granted, it’s still quite small). I’m testing the core functionality of the new application so far, and it only takes ~40s to run all the features when using ‘progress’ output (the  F..F..E  style output that test/unit does). The more verbose ‘pretty’ output is nice, but quickly fills the terminal, so very easy to miss errors as they happen until the end, and also adds processing time (10s cut off when I switched to progress mode).

At one point in development, I had to test outgoing emails. Thankfully, Cucumber has a really great addition, called email-spec by Ben Mabey. Writing a Cucumber feature for these emails was painless. Here is a snippet from one of the password reset features.

Scenario: Request Reset with correct email
  When I request a password reset for "example@example.com"
  Then I should receive an email
  When I open the email
  And I follow "Continue to reset your password" in the email
  Then I should see "Change My Password"

I’ve made a few contributions to the email-spec project, which have been included in the latest release, enabled email testing for people other than yourself  ( When they open the email… Then they should see …  etc). Makes writing tests even easier.

Other Thoughts

I did find that you duplicate a lot when it comes to adding paths because Cucumber is framework independent. For example, if you have a ‘map.resources :events‘ Cucumber won’t provide you with paths like ‘the events index’ or ‘add a new event’ automatically. Would make a handy Cucumber addition though. In addition, because Cucumber uses Webrat, which has no idea how to interact with Javascript, you still need to fall back to the slow Selenium testing if you’re application relies on Javascript.

Conclusion

Cucumber has been really great and I’m happy to keep using it. If you’ve used Cucumber before, or are going to try it, or perhaps your didn’t like it and went with something else, let me know in the comments. I’m always happy to hear other peoples experiences.

SHOW INDEX FROM table_name;

But what if you want to list all index’s from all tables in a database so you can find out what you have and where you’re missing some? If you have 20 or more tables it’ll get tiring after a while running the SHOW INDEX command on each.

Thankfully, the information is stored in another table MySQL uses called information_schema. You can get a list of all the index’s, along with the table name and field names, by running the following (change “your_database”).

SELECT table_name, column_name, index_name FROM information_schema.statistics WHERE index_name != ‘primary’ and table_schema = ‘your_database’;

That’s it. A list of what table, what field, and the name that each index applied to. Happy indexing.

Well, the testing and unstable repositories require you to update nearly your entire system, which is not ideal for just one package. But thanks to the clever maintainers, a backports repository has just what we need. ssh into your host, and make sure you have sudo access. The following instructions require root access.

Start by adding a new source to your apt-get sources. Open up /etc/apt/source.list as root, and add the following line:

deb ftp://ftp.nz.debian.org/backports etch-backports main contrib non-free

Next, we’ll install the keyring needed to install software from this server

sudo apt-get install debian-backports-keyring

Once this is done, we need to clear some space for the update command to work properly.

sudo apt-get clean

And then we can pull the information of new sources

sudo apt-get update

If the update command doesn’t work (you get warnings like ‘Dynamic MMap ran out of room’), refer to this forum post. Keep increasing the cache limit by 100,000 until it works).

Before we install, make sure you dont have a previous version (it could cause problems). Uninstall Git by running

sudo apt-get remove git-core git-svn git-doc

And you’re set to install. Because backports are disabled by default, we need to pass apt-get the -t option, to specifiy where to get it from.

sudo apt-get -t etch-backports install git-core git-svn git-doc

And once again, clean the tmp directory of apt-get leftovers, so you dont get disk full problems

sudo apt-get clean

Now, if you run ‘git –version’, you should see you have Git 1.5.6 installed. It’s not the most recent version, but its certainly better than 1.4.4.  Now you can enjoy all the new features of Git.

We’ll start by uploading that records.xml file we made in Part 1 to the server. Using an SFTP client (or scp if you use the console), copy records.xml to a folder inside the imports folder in the root of you Kete installation. The final location will look something like

/home/kete/app/your_app/imports/phpwiki/records.xml

(with the first 4 folders differing depending on your setup). Make sure you upload as the kete user (or whatever account is running your Kete installation) or Kete will have problems accessing/writing to that folder. If you can’t upload as the user, as root (if you can access that), run something like

chown -R kete /home/kete/app/your_app/imports/phpwiki

With that uploaded, lets then create a new basket to put these Phpwiki pages into. Login as an admin and at the bottom of the page, in the Administrators Toolbox, click add basket. Enter a title that will help easily recognize where these pages came from. The rest of the options should be fine if you are importing public topics, but if you exported the XML with the WILL_IMPORT_AS_PRIVATE constant in parseWikiToHtml.php set to true, then you will need to set Privacy controls to enabled at the very least. Finish creating the basket.

Now you need to set up some important synonyms for the import. These map what fields in the XML file go into the fields in your database. In the case of this import, we want pagename to be the title, and content to be the description of the topics it will be making. To set these, as an admin, click Reconfigure Site in the Administrators Toolbox, go to Advanced Settings, and then to Import. In the Description Synonyms, enter the following:

['content']

and then enter the following for the Title Synonyms:

['pagename']

If you already have synonyms in place, you may need to join then together like so:

['existingsynonym', 'othersynonym', 'pagename']

Once they’re entered, hit Save, and because you changed system settings, you’ll need to restart mongrels and backgroundrb processeses for the import to work correctly. If you’ve read this far, I’m going to assume you have a Kete installation you setup and you know how to restart your server. If not, check out the documentation at http://kete.net.nz/ .

Once restarted, we need to setup a few extended fields. In the Administrators toolbox, head to extended fields, an create three new ones with the following details:

Label: Author
Import Synonyms: author

Label: Creator
Import Synonyms: creator

Label: Last Modified
Import Synonyms: lastmodified

Leave the rest of the fields as their default. When all three are saved, the next step is to create a new topic type these topics will be imported as. Once again, in the Administrators Toolbox, click on topic types, and add a sub type of Topic. Give it a name and description that will clearly identify what these articles are. Create it, and scroll down to the section “Available form fields”. Tick all three of the new extended fields you added (Last Modified, Created, and Author) as optional, add them, and scroll down to reorder them how you want them to be shown.

Everything is now ready to be imported. Head to the basket you setup, and at the bottom, in the “Tools for basket” box select “import content into basket” and then “start new import”. After the initial instructions on how to import (which I wont explain) you’ll reach the form, where you can enter the following details:

Name of import folder: phpwiki
Delay between records: 10  (higher if you have a slow computer, lower if you have a computer with lots of memory)
Import type: Simple XML Topics
XML Path to Record: pages/page
Applicable Topic Type: [the new topic type you made]
Default Privacy Setting: [as before, make private if you exported the XML with the WILL_IMPORT_AS_PRIVATE constant in parseWikiToHtml.php set to true]

Fill in the rest of the fields to what you want, and hit Import. And you’re done. If everything was set correctly, within 30 seconds you should see the page update informing you how many records have been processed. If you set a high processing delay, it can take well over an our depending on how many records you wiki has. Once its finished, your topics will be accessible. That said, there are a few gotchas with this process.

1. Importing private topics does not, as of writing this article, update the zoom records correctly. So after importing any private topics, you will need to rebuild your zoom index’s. Check out the last 3 steps of this troubleshooting guide.
2. Links that used to work on the old wiki don’t know where to point to, so in the export process, it changes the links to searches within Kete (which it knows how to perform). So whenever you click a link in an imported topic, it will be searched for, rather than being taken directly to it.
3. The import process messes with links which contain an ampersand (which if you import as private will be all links). To fix this, you can run a set of console commands in a production console. Enter the console by running  ’script/console production’ in your Kete root directory, and copy and paste the following lines one by one (might take a while): http://gist.github.com/3359

Any question? Feel free to ask.

If you have a set of articles on a phpwiki that you can’t afford to lose, and want to move to Kete then theres a simple two part process you can follow to do just that. Over the next two blogs posts, I’m going to show you just how it’s done.

Kete can import text using a simple XML import. And in some cases, it can be as simple as exporting your mysql table to an XML formatted file, and sending it to Kete. But in the case of Phpwiki, special markup is stored in the database which Kete doesn’t know how to interpret. So while the bulk of the text would be imported, all styling and links you have will no longer work, and if you have a lot of articles, then it would get annoying to have to update each one.

So to get around this, we write our own XML export script, which takes the special syntax, runs it through Phpwiki’s transform.php file, which returns an HTML formatted string we can then use. But we have another problem. Phpwiki’s transform file is executed procedurally, which means it starts with something at the top, works its magic, and then outputs what it has at the end. It also has one function in the file, so including the file multiple times would cause problems.

file_get_contents is the answer. We have a main script that loops through the records and gets each current entry. When it comes to outputting the content, we send the title of the page we want returned to another script, which gets the contents of that page using a modified version of phpwiki’s display.php, assigns it to the right variable and includes the transform file near the end. The resulting HTML is brought back to the original script, and inserted into the XML, before it loops to the next entry and repeats the process.

Add in a few lines of code to fix search links, remove annoyingly placed Phpwiki developer comments, and escape content, and the export script works like a charm, producing an XML formatted file with the latest content which can now be imported into Kete. You can check out the finished source code of the exporter online at MediaFire.

• exportToHtml.php: http://www.mediafire.com/?dv9zug1g0rz
• parseWikiToHtml.php: http://www.mediafire.com/?yapjdul1jt0

To use the scripts, open each file and edit the defined constants at the top of the scripts. Then upload those to the base of your wiki (http://yoururl.com/wiki/ for example), and navigate with your browser to http://yoururl.com/wiki/exportToHtml.php . The resulting XML document is what you’ll use to import in the next step. Save this file to your desktop by going to File > Save page As, and name it records.xml.

Stay tuned for part 2 where I explain how to import this XML file into Kete.