Katipo Developers Blog

Today I had to evaluate whether it was worth it to use Kete’s bulk import facility to migrate an existing site’s content to Kete or whether to just have someone drag the content over page by page.

To figure this out, I wanted to know roughly how many pages along with other files were on the site. I knew that it wasn’t going to absolutely massive, so I started by grabbing all of the site’s public content with wget (the details coming from http://linuxreviews.org/quicktips/wget/):

wget -p -r --wait=20 --limit-rate=20K -U Mozilla http://the_site/

I let that run in the background while I did other work.

When it finished I wrote up a little (ugly, unDRY, but took < 5 minutes) Ruby to give me a report by file type and called it file_report.rb based on a skeleton grabbed from http://blogs.sourceallies.com/2009/12/word-counts-example-in-ruby-and-scala/:

require 'yaml'

# Change rootDir to the location of downloaded site
rootDir = "/path/to/roodDir/for/entire/downloaded/site"
raise rootDir + " does not exist" unless File.directory? rootDir

# recursively add files and directories to report_hash based on their type
def files(rootDir, report_hash)
  report_hash['directories'] = report_hash['directories'] || Array.new

  Dir.foreach(rootDir) do |dir|
    if dir != "." && dir != ".."
      dir_path = rootDir + "/" + dir

      if File.directory?(dir_path)
        puts "Processing " + dir

        report_hash['directories'] = report_hash['directories'] << dir_path

        Dir.foreach(dir_path) do |file|
          if file != "." && file != ".."
            file_path = rootDir + "/" + dir + "/" + file
            if File.directory?(file_path)
              report_hash['directories'] = report_hash['directories'] << file_path

              files(file_path, report_hash)
            else
              # add path to report_hash's entry for the file extension
              extension = File.extname(file).sub('.', '')

              report_hash[extension] = report_hash[extension] || Array.new

              report_hash[extension] = report_hash[extension] << file_path
            end
          end
        end
      else
        # add path to report_hash's entry for the file extension
        extension = File.extname(dir).sub('.', '')

        report_hash[extension] = report_hash[extension] || Array.new

        report_hash[extension] = report_hash[extension] << dir_path
      end
    end
  end
end

t1 = Time.now
report_hash = Hash.new
files(rootDir, report_hash)

puts "File type counts: "

report_hash.each do |k, v|
  puts "#{k} : #{v.size}"
end

puts "Writing complete report"
File.open('report.yml', "w") do |f|
    f.write(report_hash.to_yaml)
end

t2 = Time.now
puts "Finished in " + (t2 - t1).to_s + " seconds"

Finally, I called it with:

ruby file_report.rb

Nothing flash, but handy to have when you need it. I saved myself a lot of clicking around on their website.

I’ve moved from MacPorts to Homebrew which includes a recipe for installing MongoDB. After installing Homebrew, just run this as your normal user:

If you prefer to store your MongoDB data all under your home directory, you might find Mislav’s gist suits your needs instead:

http://gist.github.com/265272

If you prefer installing from source, check out this post:

http://shiftcommathree.com/articles/how-to-install-mongodb-on-os-x

Enjoy.

The OPACs at HLT have been failing one by one, so some replacements are in order.  The previous OPACs ran Mozilla in kiosk mode on a hand modified Debian install sufficiently rickety that after ~4 years I didn’t really want to touch it.

Somewhat surprisingly, the task (taking some hardware and netbooting it into a teenage-vandal-proof web kiosk) didn’t appear to be a solved problem – there are still a lot of people rolling their own, I’ve done that once, I didn’t really want to do it again.

Webconverger almost fits the bill, with a couple of caveats: (more…)

The other day I was asked how much disk space a particular Kete project’s production database used. I didn’t have an answer, so I started searching for one.

In my case, the db software is MySQL and getting a report about various aspects of a db’s tables boils down to this query:

The output of this query has way more information than I need, plus to get the total number of bytes used you have to add all the values up anyway, so I decided to make a rake task to make this repeatable and return something succinct.

I have added the new rake task to the Kete app’s codebase. If you aren’t a Kete user, but need this functionality for your Ruby on Rails app, you can find it here:

http://svn.kete.net.nz/projects/kete/trunk/lib/tasks/db-disk-usage-report.rake

Note there is a formatting method taken from the Rails Helpers, I would love to not duplicate this definition. If anyone has a recommendation for the best way to pull this method into this rake task, I would appreciate it. Probably a simple include statement, but I didn’t get around to figuring out the correct incantation.

I would also love to hear how best to make this handle other database software, for example PostgreSQL, too.

Cheers,
Walter

As previously mentioned, I have written a guide on how to set up Kete (also applicable to other Rails apps though Kete includes some extra required software and niceties to make it easier) for Development on Mac OS X. This guide also includes best practices for Deployment. So the following guide can be seen as the first half of the story:

http://kete.net.nz/documentation/topics/show/16-creating-a-kete-development-environment-on-mac-os-x

But what about the host that you are deploying to? I cover that for Debian Etch in the following guide:

http://kete.net.nz/documentation/topics/show/15-preparing-a-debian-etch-host-to-be-deployed-to-for-kete

You might also be curious what software Kete requires. Here’s a breakdown:

http://kete.net.nz/documentation/topics/show/19-technical-requirements

Cheers,
Walter

I’ve added a guide over at Kete.net.nz for how to install Kete for development on Mac OS X. You can find hit here:

http://kete.net.nz/documentation/topics/show/16-creating-a-kete-development-environment-on-mac-os-x

If anything doesn’t work for you, please let me know.

Cheers,
Walter

I’m going to outline how to set up Aquamacs for my IRC requirements with the built in ERC emacs IRC client so that it automates joining the server and channels I want, with some other bells and whistles (the bell part, literally!). All of these steps have .emacs file equivalent, so you could use it as a starting pointing point for figuring out how to set them, even if you don’t use Aquamacs (Mac OS X GUI emacs).

Here are the steps:
(more…)

People are moving away from managing their Rails plugins with svn externals towards Piston’s easy vendor branch management approach.

But what if you want to use Piston’s easy version of Subversion’s vendor branch management for dealing with the code for an entire Rails application based on an open source project like Mephisto, Collaboa, or in my case Kete? (more…)

From a howto on Robby on Railshere. You’ll want to read http://www.rubyinside.com/advent2006/12-piston.html for more commands and reasons to go this route.
(more…)

I have a host where Nginx listens on port 80 and dishes out requests to various apps depending on what their host name is. It also does appropriate load balancing. Primarily I use this for hosting various instances of Kete Ruby on Rails apps, but I also use it for supporting the subversion repository under Apache and a few other RoR apps, too. I wanted to setup up web log statistics for all of the above and I wanted to do it in such a way that it’s easy to add another web app to view statistics for.

Awstats is wicked easy to setup if you live in the LAMP world, but I could find a decent tutorial that covered my setup. So here goes… (more…)