Computation

I recently became interested in Clojure, a (mostly) functional lisp that runs on the JVM. While reading the documentation and watching the screencasts, I wanted to start trying out some of the stuff I was seeing. I use Emacs for code, on Windows, Linux, and Mac OS, and it is particularly nice for dynamic languages that provide a REPL. Here's the super easy way to get started with Clojure on Emacs.

1. Install a recent version of Emacs, 22 or 23, if you haven't already got one. I use Carbon Emacs on OS X.
2. Follow the instructions here: Install the Emacs Lisp Package Archive (ELPA) script
3. Get a list of available packages (optional, but interesting): M-x package-list-packages
4. Install the Slime and Clojure packages:
• M-x package-install clojure-mode
• M-x package-install clojure-test-mode
• M-x package-install swank-clojure
• M-x package-install slime
5. Run Clojure in SLIME:M-x slime. You should be asked if you want to download and install Clojure. Say yes, and let it do it's thing, and you will eventually see a buffer that looks like this:

; SLIME 20091016
user> 
user> 

6. Read stuff on clojure.org, watch the screencasts on clojure.blip.tv, and have fun!
7. Try out ants.clj as detailed at this other not so easy Clojure+Slime+Emacs guide. Skip down to the "Test Configuration" section.

For more information about how Slime works, see this post by Bill Clementson.

This Emacs+Slime+Clojure video on Vimeo may also be interesting, although dated.

Today I've posted Phred, the 3d FDTD program I wrote for my M.A.Sc, to Google code. In some ways it is very simple: it only supports uniform grading for instance. The way the updates are handled is not very flexible. The testing is pretty primitive. Other things about it are still really cool: it runs on a variety on super-computers, or at least it did five years ago. It uses Python as a scripting language for problem set-up. It implements the Drude model for modelling some kinds of metal. It's a good example of my skills as they were five years ago. Despite cringing at some of this code, I'm still pretty proud of it.

You can find the Phred Google Code project page here.

Cool stuff about Phred:

• Runs on a variety of machines:
• Many variants of Linux
• IBM eServer BladeCenter HS20 (WestGrid's Glacier)
• IBM pSeries SMP (WestGrid's Cortex)
• HP AlphaServer running Tru64 (WestGrid's Lattice)... I think. There might have been compiler issues.
• SGI Origin (WestGrid's Nexus)
• IBM SP Nighthawk-2 (UVic's Minerva)
• Problems are scripted in Python
• Drude model for simulating dispersive materials such as gold and silver.
• Uses a hybrid MPI/OpenMP model for parallel computation. I found that MPI by itself generally gave better performance, even on SMP machines.

I ran the majority of my large simulations on Nexus and Glacier, and smaller simulations on Debian Linux, so those are the best tested. However, things have come along way in five years, and I'm not sure if everything would still compile given various software updates than may have been rolled out.

There are a lot of rough edges and unfinished things in the code. I was experimenting with template meta-programming for the updates, for instance. Template meta-programming is interesting, but because it's a side-effect of a language feature designed for something else it is hard to use. The error messages are particularly unhelpful.

My goal in releasing this is mostly just to preserve it for my own reference, to ensure that it is not lost when the hard drive it was on is disposed of or dies. Open source is more than just the code. It takes a lot of work to build up a community, get people interested, accept patches, and so on. I don't have time to give this project that kind of love. Perhaps someone else will be interested in picking it up and taking it over, but that seems unlikely. There are already many other actively maintained open source FDTD packages out there.

I'm using Cygwin and X11 on Windows 7, and I'm pretty happy with it so far. The newest version of Cygwin integrates nicely, and 7 seems pretty polished and much nicer overall than XP (I never really used Vista).

The one problem I've been having is related to the meta key in XTerm. I'm a heavy user of the Emacs movement keys, so M-b, M-f, M-a and friends see heavy use. However, in the default configuration, I get garbage and strange behaviour.

I finally found the fix in an old blog post from 2005. To summarize, the fix is to ctrl-left click and make sure that "meta sends escape" is selected. This can be made permanent by adding this to your .Xresources file:

XTerm*metaSendsEscape: true

To load that change without restarting X, run xrdb -override .Xresources.

Now to figure out what is wrong with X11 on Snow Leopard!

UPDATE: If adding the above your your .Xresources file does not work, try adding it to .Xdefaults instead. Alternatively, add xrdb -merge ~/.Xresources to your .xinitrc file.

I was trying to set up netatalk to serve files off my linux box to my macs, but Leopard was giving a cryptic error message whenever I tried to connect to the server. Turns out the that the default netatalk package that comes with Ubuntu doesn't have SSL support enabled, which Leopard requires. I finally found a blog entry that shows how to build an SSL enabled netatalk package. Works like a charm now!

AFP replaces SSHFS and Macfuse for me, as I couldn't get the volumes to mount read/write after upgrading to Leopard. AFP is really more suitable anyway.

I've been trying to upgrade my MacBook Pro to Leopard for the last few days. The trouble is, the installer refuses to detect any of the partitions on my hard drive. It doesn't even show the drive with a exclamation mark and tell me I need to repartition, which seems to be fairly common. The Disk Utility and command line tools in the installer don't see the partitions either, so it seems to a problem at a more fundamental (kernel?) level. Tiger works fine, as does the installer for Tiger. From searching the inter-tubes, it appears that the problem is related to my bootcamp partition. The Leopard installed seems to require a specific partition layout in order for it to detect the partitions, which seems massively stupid. Especially when bootcamp installations are probably fairly common on Tiger machines.

I tried a couple of things already, such are resetting the protective MBR to contain only the EFI protection partition, but that had no effect. Since I have rEFIt installed, I wasn't worried about trashing the MBR, since rEFIt can recreate it from the GPT.

A guy on this thread about the Leopard Installer disk detection problems suggests that the installer requires the GPT itself to be set up in exactly the right way. It requires either a single HFS+ partition of any size, or a single HFS+ partition followed by a "basic data" partition, as long as the sum of the two partition sizes is exactly the size of the disk. My disk appears to fit the second case... Except it doesn't, because there is a gap between the HFS+ and FAT32 partitions, which can be seen with the gpt tool:

thunderstorm:~ mch$ sudo gpt -r -v -v show /dev/disk0
gpt show: /dev/disk0: mediasize=80026361856; sectorsize=512; blocks=156301488
gpt show: /dev/disk0: PMBR at sector 0
gpt show: /dev/disk0: Pri GPT at sector 1
gpt show: /dev/disk0: Sec GPT at sector 156301487
      start       size  index  contents
          0          1         PMBR
          1          1         Pri GPT header
          2         32         Pri GPT table
         34          6         
         40     409600      1  GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B
     409640  102760448      2  GPT part - 48465300-0000-11AA-AA11-00306543ECAC
  103170088     262144         
  103432232   52869216      3  GPT part - EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
  156301448          7         
  156301455         32         Sec GPT table
  156301487          1         Sec GPT header

So, there is a 6 sector gap between the table and the HFS+ partition, and a 262144 sector gap between the HFS+ partition and the FAT32 partition. That second gap is probably what is messing up the Leopard installer.

Here's what diskutil has to say about the disk:

thunderstorm:~ mch$ sudo diskutil list disk0
/dev/disk0
   #:                   type name               size      identifier
   0:  GUID_partition_scheme                    *74.5 GB  disk0
   1:                    EFI                    200.0 MB  disk0s1
   2:              Apple_HFS Thunderstorm       49.0 GB   disk0s2
   3:   Microsoft Basic Data HELL               25.2 GB   disk0s3

Yes, my Windows partition is named HELL. Here is my MBR, after I trashed it so that it only contains the protective GPT partition:

thunderstorm:~ mch$ sudo fdisk /dev/rdisk0
Disk: /dev/rdisk0       geometry: 9729/255/63 [156301488 sectors]
Signature: 0xAA55
         Starting       Ending
 #: id  cyl  hd sec -  cyl  hd sec [     start -       size]
------------------------------------------------------------------------
 1: EE    0   0   2 - 1023 254  63 [         1 -  156301487] 
 2: 00    0   0   0 -    0   0   0 [         0 -          0] unused      
 3: 00    0   0   0 -    0   0   0 [         0 -          0] unused      
 4: 00    0   0   0 -    0   0   0 [         0 -          0] unused      

The next step is to back up my boot camp partition before taking the FAT32 partition out of the GPT... I'm hoping that that will let me install Leopard, after which I can attempt to restore the GPT entry for the FAT32, and hopefully everything will work.

The "My account" page on the Calgary Public Library site uses your library card number and a pin number to authenticate you. However, both form fields are of type "password", which means that Firefox won't save the username / password information. Since the site seems to randomly log you out, it becomes a pain in the ass to get out the library card over and over again to type it in by hand.

This Greasemonkey script simply changes the library card barcode number field from a password type to a text type, which allows Firefox to remember the contents. Unfortunately, when Firefox fills the form, it places the "password" (your pin) in the first password field it finds... since the autofill happens before the Greasemonkey script runs. The barcode isn't autofilled anywhere, so it is necessary to embed it in the script itself. Or you can type the first number in the barcode field and it will autofill it then.

NOTE: after using the CPL website a while, I noticed that the original version of this script was filling in random form fields. It needed to be modified to look for some unique characteristic of the sign in page. Since the CPL website uses a single JSP and GET URL parameters to navigate all the pages, one can't use the greasemonkey include/exclude URL thing.

// ==UserScript==
// @name           Calgary Public Library Login fixer
// @namespace      http://spacemonkeys.ca/greasemonkey
// @description    Changes the password box to a text box so 
//                 that firefox can remember the form fields. 
// @include        https://catalogue.calgarypubliclibrary.com/ipac20/*
// ==/UserScript==

var inputFields = document.getElementsByTagName('input');
var barcode = inputFields[9];
var barcodeLabel = barcode.parentNode.parentNode.childNodes[0].childNodes[0];
var pin = inputFields[10];
var pinLabel = pin.parentNode.parentNode.childNodes[0].childNodes[0];

var mycardNumber = "xxxxxxxxxxxxxx";

// Firefox autofills the pin into the barcode field, probably because
// that is the first password field it sees.  The barcode itself is lost.
if (pin.name == "sec2" 
    && barcode.name == "sec1" 
    && barcodeLabel.data.search("Library Card Number:") != -1 
    && pinLabel.data.search("Last 4 numbers") != -1)
{
    var temp = barcode.value;
    barcode.value = mycardNumber;
    pin.value = temp;

    // Make the barcode number a text field, so that Firefox will be able
    // to remember the barcode/pin combo. 
    barcode.type = 'text';
}

It looks like the CPL is in the middle of overhauling their site, so maybe the need for this script will go away.

If anyone can tell me how to get Firefox to redo the autofill after changing the barcode field to text, that would be great!

I'm often asked how many lines of code there are in some project I'm working on. I think it's the least useful metric of any codebase, but people always seem interested in knowing. I'm not sure if they are more impressed by a small number or a large number...

Since it comes up, here's a one liner to count the number of lines of code.

find . -name "*.cc" -or -name "*.hh" -exec wc -l "{}" ";" | gawk '{nlines = nlines + $1} END {print nlines}'

If you want to have multiple OS's on your USB drive, you can install GRUB. I haven't actually installed multiple OS's, but GRUB does work:

1. Mount the drive somewhere, say /media/usb.
2. Copy the grub files to the flash drive: mkdir -p /media/usb/boot/grub, then cp -R /boot/grub/* /media/usb/boot/grub.
3. Change the device map for the flash drive: echo "(hd0) /dev/sdb" > /media/usb/boot/grub/device.map
4. Run grub using the new device map: grub --device-map=/media/usb/boot/grub/device.map
5. Install grub on the flash drive: root (hd0,0), then setup (hd0), and quit.
6. Edit the /media/usb/boot/grub/menu.lst file so that it will load FreeDOS by default:

   
         default 0
         timeout 10
   
         title FreeDOS
         root (hd0,0)
         chainloader +1
         makeactive
         boot
   

7. Unmount and test in qemu, or on a real machine.

It is sometimes necessary, even in this day and age, to use DOS or some other simple operating system to directly access hardware to perform tasks such as BIOS upgrades. Floppies, the traditional boot media for quick and dirty tasks such as this, are almost extinct. CD's are cumbersome to use and change. It is therefore handy to use some other portable external bootable media for jobs such as this. Enter the bootable USB key.

FreeDOS is a handy open source alternative to MS-DOS, but it only comes as a CD image that must be burned to a CD and then installed to a hard drive in order to be usable. This makes it a bit cumbersome to install on a USB key.

I took a somewhat circuitous route to arrive at these instructions, which included installing FreeDOS in a VMware Fusion virtual machine, copying that virtual disk to a flat image file, moving that image file into a Ubuntu virtual machine, byzantine manipulations of a USB flash drive with dd, and learning more about FAT and MBR than I ever wanted to know. Hopefully the distilled instructions below are a little easier.

I assume that you have a Linux capable of running QEMU (there is a version of QEMU for OS X). You don't need to use QEMU... but it makes it easier to try different things are repeat the process while still doing other things, since you don't have to reboot all the time.

1. Download the FreeDOS installer ISO from FreeDOS.org.
2. Partition your USB flash drive as you please. You should have at least one FAT32 primary partition on which you will install FreeDOS.
3. Run qemu-launcher. Set it to boot from CD-ROM, and set the CD-ROM path to point to the FreeDOS ISO you just downloaded. Set hard disk path to point at the raw flash device (something like /dev/sdb... find out from dmesg).
4. Launch qemu. It should boot the FreeDOS installer. Install FreeDOS on the "hard drive". I recommend installing just the base set of packages, unless you want to experiment with networking, etc in FreeDOS.
5. If you don't need multiboot, you are done.
6. Close QEMU, unplug the flash drive, and try to test it on a real system. If you have trouble, like it says that no OS is found, it may be necessary to run QEMU again in the same way as before, except instead of using the installer, use the live CD and run fdisk /mbr and sys c: to make sure the drive is fully bootable.

If you want to use multiple OS's on the flash drive, then you can install grub and set it up to boot each one. But that's for a later post...

I often tried to use bash's for loops to do line by line processing, but it's alway been a pain in the ass. An example. I have a list of files in a text file, and I want to copy each file to some other location. The naïve method:

for F in `cat names.txt`; do cp $F /new/path; done

Unfortunately, names.txt looks like this:

This filename has spaces 1.txt
This filename has spaces 2.txt
...
This filename has spaces N.txt

Bash breaks each line on the space, so the for loop doesn't copy each complete filename. If there are no spaces or tabs in the filenames, then the for loop works ok. The right solution is to use the read command and a while loop:

cat names.txt | while read FILENAME; do cp "$FILENAME" /new/path; done

Update per comment below: added quotes around $FILENAME.

Thanks electricmonk!.

Apple recently released a new version of the iWork office suite, which now includes a spreadsheet application, Numbers. One thing that annoyed me about Numbers was the data and time formats. I like to use ISO 8601 (YYYY-MM-DD) because I can never remember which is the month and which is the day in the MM/DD/YY or DD/MM/YY format, which is different between the US and Europe. Numbers doesn't have the ISO 8601 format, and doesn't allow you to specify a custom format. Which is kind of annoying, but it is the first release.

Fortunately, someone figured out how to do it. Here is how to do custom date formats in Numbers. . I Haven't tried it out yet though.

Now if only I could figure our how to add functions.... I need WORKDAY() for my schedules....

I replaced this keyboard from Tamara's iBook this morning with one I got off of eBay. All three screws that hold the body to the frame in the battery compartment fell out when I took the battery out, which was a little surprising. Makes me wonder how many other screws have worked themselves loose. The memory shield screws were still nice and snug though.

I recently upgraded a Dapper Drake machine to Edgy Eft. After the upgrade, the machine would not longer boot. A kernel panic occurred due to a crc error loading the initial ramdisk, followed by the dreaded "unable to mount root fs on unknown-block(0,0)" message.

I actually fixed this once before and forgot how I did it, so I'm documenting it for prosperity. The crc error suggests that the initrd file, which holds modules and scripts necessary for getting the system up to the point where the root filesystem can be mounted, was corrupt. It is possible that running update-initramfs -c -k 2.6.17-10-generic would have solved the problem. However, I also appended the names of the IDE modules and the VIA IDE chipset module to the /etc/initramfs-tools/modules file, so that they would be copied into initrd and loaded before the root filesystem had to be mounted.

The machine booted flawlessly with the new initrd installed.

I use EMACS for the vast majority of editing, but the support for word wrapping has always been a bit of an annoyance. Especially when writing stuff to go into web forms. By default, the only way to wrap words so that lines don't run off the edge of the end of the window is to use auto-fill-mode, which inserts newline characters at the end of the lines. This is exactly what is required for text files that are expected to be read with dumb pagers like less or more, but it can result in erroneous line breaks in things like web forms. An additional annoyance is that auto-fill-mode only inserts line breaks automatically when the cursor reaches the end of the line, so if you go back and edit in the middle of a line, you have to type M-q to get the line breaks back into the right place.

Enter LongLines mode. LongLines solves both of these problems by automatically inserting newlines where ever they may be required for word wrapping, but strips them out when saving or copying, so you don't see extra newlines in your forum posts.

yay!

I am doing a presentation today at work, and when I was digging through my desk to find the DVI->VGA converter that came with my Mac, I came across the little remote control that also came with it. I thought it would be cool to put it to good use by using it to flip slides, rather than running up to the keyboard all the time.

I use NeoOffice (a port of OpenOffice.org to OS X), not because it is pretty (it's not), but because it cheap and does a good enough job of emulating Microsoft Office. I haven't tried Keynote, but it sure looks slick, and it turns out it's not as expensive as I had heard (upwards of $300!). Turns out it is only $89 for iWork '06, which includes a word processor (Pages). Not that I ever use a word processor (hooray for text files and typesetting software!). I think a demo of iWork was installed on my Mac when I got it, but it seems to have been deleted....

Anyway, to get the remote control to handle flipping slides in NeoOffice, I first tried iRed Lite, because it is free. However, the interface is quite amateurish, and it isn't able to control NeoOffice Impress, although it can apparently control OpenOffice Impress. It is also necessary to manually select the "profile" to use; it doesn't automatically control whatever program is in the foreground. One good thing is that you can press and hold the menu button to enable or disable iRedLite, so that you can easily get back to the default Front Row behaviour.

I then tried Remote Buddy. Wow. For about $14.47 CAD, you get a much slicker interface, the ability to automatically control whatever is in the foreground, AND, out-of-the-box support of NeoOffice Impress. I'm just taking advantage of the 60 day demo, but if I were to use the remote a lot more (unlikely), this might be a worthwhile purchase! The only bad thing is that because the menu button always brings up the remote buddy menu, and because holding the menu button is often assigned to a task in an application, it is necessary to use command-esc to bring up front row. Which is not quite as simple and just pushing the menu button on the remote. Oh well.

There are a few other remote control programs out there, such as Mira and Sofa Control. Amit Singh also has more technical information on the IR device and how to use it, along with a little irdaemon that can be used to control applications.

Now that I've wasted an hour playing with the remote control and watching movie trailers with Front Row, I guess I should finish working on my presentation...