Home Play Pinja Bobbity flop

A blog about Ruby, Rails and other Tech. Mostly.

To contact me email: (this is a spam protected address - please do email me because comments are not implemented in this blog). There is also an XML/RSS feed.

9th Feb 2009, 12:00amPermalink
Finding the last memory chip in Toy Bot Diaries 3

I don't know what the developers were thinking, but the last, impossible to find, memory chip is in fact hidden in the blackness of an area you normally wouldn't want to fall into (or you die).

It's in "Down in the Dumps".

Get to here:

Take the catapult, you'll land over here:

Walk to the left:

And just walk off the end. If you go the right speed you'll hit the memory chip on your way down just before you die.

I wonder if the chip can be got without dying?

[Edit] The very day after I posted this, IUGO updated the app, and fixed this issue. Now there is a brick to stop the memory chip falling in the soup, and it can be easily retrieved without and danger.

That's better

20th Oct 2007, 7:55pmPermalink
This blog has moved

My all new blog: New blog software from Wordpress, and a new domain name pennysmalls.com.

I gave the blog a new domain all of its own as it's focus is mostly rails and associated tech, and not much to do with my family and personal stuff.

The RSS feed is here.

Yes, the layout is dull, I'll get to it.

Most of the recent posts are already there, I'll keep this blog around as an archive of the older stuff.

17th Oct 2007, 8:01pmPermalink
Adding an Expires header with apache for Rails

We have a problem - we really do. Each time a user requests a page they have to make 50 http requests just to get back a "Not modified" message from the web server. Their browser is asking about every little image and css file and so on. Every one on the page. Those messages are small, but they add up. And as we know, "make fewer HTTP requests" is Steve Souder's number one rule for speeding up your website.

What we need to do is to add an Expires header to our static content - one that gives a time a long way in the future so that the browser knows that the content should stays in the cache -and not be fetched again, or even checked (if it has been updated) from the server. But what if we want to change a css file or a js file? The user will get the old one from their own cache. No good.

Well Rails has a mechanism to prevent this - it adds a 10 digit number on the end of each url (as generated by image_tag, stylesheet_link_tag, or javascript_include_tag for instance) in the query string. That number is based on the file modification time - so when the file is updated then the URL will change. It's like having a remote way to expire the item in the browser's cache.

So it's simple right? Just turn on mod_expires in Apache? Not so fast. What about those images and items that do not have the query string? We don't want to send an expires header for those. Definitely not. (And you will likely have some - for instance images that are referenced from your css files.) If you do, the user is stuck with their cached version even if you change the file on the server.

So we need some way of selectively turning on the expires header in apache.

One way [Danny Burkes] (look at the update at the bottom) is to segregate by directory what you want to expire and what not. But this seems a bit clumsy to manage. (Also, side issue, I am not totally convinced that the munging of the urls provided by the plugin is needed - at least section 13.9 of the HTTP 1.1 spec seems to suggest that content with query strings will be cached fine if an explicit expires header is given.)

Actually you can do what you need with one symbolic link and some apache magic. The obvious magic won't work - you can't detect what's in a query string using a LocationMatch or FilesMatch container. But we can get around this with a rewrite rule, and a directory container. This is from my apache httpd.conf file:

  # add something we can do a directory match on
  RewriteCond %{QUERY_STRING} ^[0-9]{10}$
  RewriteRule ^(.*)$ /add_expires_header%{REQUEST_URI} [QSA]

  # the add_expires_header directory is just a symlink to public
  <Directory "/path/to/rails_app/public/add_expires_header">
    ExpiresActive On
    ExpiresDefault "access plus 10 years"
This detects those query strings (we assume you don't use 10 digit query strings for anything else), and adds a directory on the front of the path.

We use this as something we can detect in a Directory container. And in there we turn on the expires header.

We need one more thing:

cd /path/to/rails_app/public
ln -s . add_expires_header
The symbolic link doesn't go anywhere, and that's just what we want. All the images and css files and whatnot will be found in their usual places. It's pretty unobtrusive -you don't need to change anything in your app to start to benefit from the expires header.

It's a big benefit - we have literally gone from 50 HTTP requests per page to about 16. And with some tweaking we'll get it down more - some of those are from references to images in css, but a few are due to us not using urls generated by rails for static content. And we can fix those.

13th Oct 2007, 10:03amPermalink
Benchmarking at the method level in rails controllers

We have a reasonably complicated set of logic inside our controllers that is able to take an incoming url and serve the right content depending on 3 different parts of the url in various combinations and permutations. There are quite many DB queries required, and sometimes it goes slow.

I wanted to find where the system was spending its time, and the DB query logging provided by ActiveRecord was not enough to show me where the performance needed tuning.

The rails benchmark method to the rescue. And a little meta-programming. I added this to application.rb:

    @@added_methods = []
    def self.method_added(id)  # wrap a benchmark around all controller method calls
      name = id.id2name
      if name =~ /^old_(.*)$/
        return if @@added_methods.include?($1)
        return if @@added_methods.include?(name)
      @@added_methods << name
      alias_method "old_#{name}".intern, name.intern
      eval %{
        def #{name}(*args, &block) 
          logger.info("*=*=*=*=*=*=*=*=* #{name} START")
          self.class.benchmark("*=*=*=*=*=*=*=*=* #{name}", Logger::DEBUG, false) do
            send("old_#{name}", *args, &block)

It's important to add this pretty much at the top of application.rb, or at least before all the methods you want to time.

Method_added gets called every time a method is added to the class. As each method is added we make a note of it in our array @@added_methods, and then alias it with the prefix "old_". Then we define a new method adding the logger calls and the benchmark call.

The output looks like this:

*=*=*=*=*=*=*=*=* save_url_history START
*=*=*=*=*=*=*=*=* save_url_history (0.00004)
*=*=*=*=*=*=*=*=* missing_page_catcher START
*=*=*=*=*=*=*=*=* get_categories START
*=*=*=*=*=*=*=*=* get_categories (0.00028)
*=*=*=*=*=*=*=*=* get_continents START
*=*=*=*=*=*=*=*=* get_continents (0.00004)

Also you will get all your regular debug level logging in between these new lines in your log.

Note that the @@added_methods array is important because both alias_method and the eval'd def result in a call to method_added - so we use the array to prevent infinite recursion.

Just run rails up with the EXTRA_BENCHMARKING environment variable set, and you are away:

EXTRA_BENCHMARKING=1 script/server -e production
I like to use production mode so I can see the effect of the fragment caching. But you'll get valid results in development mode too.

13th Oct 2007, 9:41amPermalink
Rails logging - config.log_level has no effect

Curiously adding config.log_level = :debug to environment.rb appears to have no effect on the logging performed by ActiveRecord. To get to see the detailed logging and benchmarking of your database queries in production mode you need to place this after the Rails::Initializer.run block in environment.rb:

ActiveRecord::Base.logger.level = Logger::DEBUG

9th Oct 2007, 9:37amPermalink
Rails testing - two gotchas

Gotcha 1

Failing tests. We'd like them to pass of course. But there is something strange going on - the error is not a test error, it seems to come from rails itself:

  1) Error:
NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.[]
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.4/lib/active_record/fixtures.rb:498:in `orders'
    ./test/unit/order_mailer_test.rb:22:in `test_searcher_email'

1 tests, 0 assertions, 0 failures, 1 errors

What does it mean? The clue here is that the error comes from fixtures.rb. The other thing to note is that I have a setup in my test:

  def setup
    ActionMailer::Base.delivery_method = :test
    ActionMailer::Base.perform_deliveries = true
    ActionMailer::Base.deliveries = []
Which should be fine, but there is this nasty error. Let's try something in the setup method - we'll add a call to super there:
  def setup
    ActionMailer::Base.delivery_method = :test
    ActionMailer::Base.perform_deliveries = true
    ActionMailer::Base.deliveries = []
And the result:
Finished in 0.160284 seconds.

1 tests, 1 assertions, 0 failures, 0 errors
Great, that's fixed it. But why? Well it took me a while to find it. Have a look in /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.4/lib/active_record/fixtures.rb - there is your culprit at line 554:
     alias_method :setup, :setup_with_fixtures
So the rule is, if you override setup you need to call super otherwise you don't get your fixtures set up properly.

Gotcha 2

Ever tried to test your routes? You should. But if you have routes that depend on what's in your database you've got a problem. Why? Because the routing is initialised before any fixtures are loaded. In fact none of the test related code is loaded at that point. So your tests just won't work.

I don't know a good way to fix this. In the end I opted for a fudge at the beginning of routes.rb:

  if ENV['RAILS_ENV'] == "test" && Category.count == 0
    Category.create!(:name=>"Families and Friends", :alternative_name=>"Family")
And I have a fixture that contains the same information. This isn't really DRY, but I don't really see a good way to do this. Maybe we could read the fixtures file ourselves and get the data from there.

But at least the tests run ok now.

3rd Oct 2007, 10:35amPermalink
debug1: Remote: No xauth program; cannot forward with spoofing

I needed to install X on a remote fedora machine, and then use my local machine as the X server over ssh. Compiling & installing X went smoothly, but I got the following error when running ssh -X (with -v for debug output) to the remote machine:

debug1: Remote: No xauth program; cannot forward with spoofing
I figured /usr/X11R6/bin needed to be in the path, but modding the path in .bash_profile, /etc/ssh/sshrc, or similar places had no effect.

It turns out that sshd has a hard wired idea of where xauth is supposed to be:

$ strings /usr/sbin/sshd|grep xauth
No xauth program; cannot forward with spoofing.
So there you are - the solution is simple:
# ln -s /usr/X11R6/bin/xauth /usr/bin/xauth

1st Oct 2007, 12:02pmPermalink
Why Ruby is not my favourite programming language

Actually it is my favourite. But the title comes from this article (which itself was inspired and copied from the famous paper about pascal by Brian Kernighan).

On with business though - Ruby has some serious wrinkles that produce counter-intuative and plain strange behaviour. One might say that they violate the Principle of Least Surprise. But POLS is somewhat personal - surprising to whom? (just me maybe!)

Surprising result #1

=> 1
a if a=b
NameError: undefined local variable or method `a' for main:Object
        from (irb):2

This is odd indeed. It seems that the Ruby parser thinks that a is a method at the beginning of the line. But when it parses a=b it will know (and remember) that it is a variable. However, when the condition passes and it comes to execute 'a', it does not apply its new found knowledge about the nature of a - it still goes for the method. And hence the error.

Surprising result #2

c=1 unless defined? c
=> nil
=> nil

This is a bit easier to explain. Once the parser has seen c=1 it enters c in the symbol table, it knows that c exists as a variable. Hence it is defined (as nil). So c=1 is never executed.

Surprising result #3

class A
  attr_accessor :a
  def b
    puts("a is #{a}")
    puts("@a is #{@a}")
=> nil
c = A.new
=> #<A:0x32fb7c>
a is 1
@a is 0

No mystery here really - it's a nice gotcha though - a= does not call self.a=, it just assigns a local variable. It is just a little confusing unless you think carefully.

Surprising result #4

[0.0, 1/0.0].sort
=> [0.0, Infinity]
[0.0, 0/0.0].sort
ArgumentError: comparison of Float with Float failed

Thanks to Evan Weaver for pointing this out to me. This is due to 0/0.0 returning the value NaN (as opposed to 1/0.0 which returns Infinity). And the comparison with NaN by sort gives the strange error message.

Let me know if you spot any more oddities.

28th Sep 2007, 12:06pmPermalink
Could not find rails (> 0) in any repository

This is with a brand new installation of ruby (./configure, make, make install) and of rubygems (ruby setup.rb). Odd error message to get really, rails is certainly there .

This helps, but actually all I needed to do was to simply say

gem update
... and then ...
gem install rails --include-dependencies
...works as it should.

24th Sep 2007, 2:45pmPermalink
Full text search systems

A quick survey of the available open source systems for text search

22nd Sep 2007, 10:58pmPermalink
Obfuscated ruby - ETA interpreter updated

Just a quick pointer - I had some correspondence resulting in some updates to my by now ancient obfuscated ETA interpreter.

The idea is to write the shortest interpreter for the ETA programming language possible. Previously I had got the program down to 227 chars of tight and obfuscated Ruby.

With these updates, either 4 or 16 chars have been knocked off the old best effort, depending on whether you want warnings or not. And both the newer versions are compatible with Ruby 1.8.6, which is a good thing seeing as the older versions were failing to run.

Although obfuscated programming is on the whole a fairly useless pastime, one certainly learns all the strange little creepy corners of a language. And Ruby has a few. (And not just this). I'll explain more in a later post.

21st Sep 2007, 6:14amPermalink
IOCCC 2006 winning entries

I am delighted to say that both my entries for the 19th IOCCC qualified for a prize.

The first was a solver for the Bedlam cube. Here are the remarks, the makefile and the program itself.

The second was a one liner digital clock program. Here are the remarks, the makefile, the program itself, and finally the script for running it.

You can check out my previous entries here (2004) and here (2005).

This year the winners are being presented again at the Vintage Computer Festival on the 3rd and 4th November 2007 in Mountain View, California. It was a great event last year, and I'll go there again if I can this time. I recommend it if you are nearby.

31st Aug 2007, 3:19pmPermalink
xtorrent 1.1 - released today

Now then, there is this very nice torrent client for the mac called xtorrent (well, it's more than just a client). And there happens to be a new version of it that you can read all about over here.

Go have a look. And see if you can guess why I wrote about it :)

Truly, all Dave Watanabe's stuff is good though, & I have been using Newsfire as my main newsreader for a long time.

28th Aug 2007, 12:34pmPermalink
Caught at the Storyville


9th Aug 2007, 8:50amPermalink
Rails: wrapping content from a helper

The form_tag method in ActionView uses a nice trick to wrap a block of content in form tags, and you can wrap content the same way yourself.

You want to write something like:

<% yellow_boxed do %>
some text and html and stuff
it's probably only worth doing it this way
if this bit is longer than a line or two
<% end %>
This reads in a very natural way. To do this, I have a helper method called yellow_boxed:
  def yellow_boxed(&block)
    concat(render(:partial=>"yellow_boxtop"), block.binding)
    content = capture(&block)
    concat(content, block.binding)
    concat(render(:partial=>"yellow_boxbottom"), block.binding)
This technique can be used in any situation where it is logical to wrap content.

Note that I render a partial for the top and the bottom of my yellow box. The partials just contain table, tr and td tags and so on, but I wanted to keep the html strings out of my helper. But for short lengths of html it's not a problem, you can put text strings there instead of the much more expensive calls to render.

Previous notes