Author Archives: Graeme Mathieson

Today I Learned: Vim command line fu

Posted on by
Reply

OK, so this was “Thursday I learned” but I figured I should write it down before I forget again. One of my frustrations with Vim is better learning to use the command line. In this particular instance, I had been searching for a phrase inside a single file and I wanted to instead search for it with git grep (using :Ggrep from the awesome fugitive.vim). Of course, I didn’t want to retype the thing I’d already been searching for. Turns out there are a couple of viable options:

  • ctrl-r / will insert the last search pattern. So, having searched for something within a file and now wanting to search for it throughout the repository, I could do :Ggrep ctrl-r /<cr>. I think I want to turn that into a shortcut of some variety…

  • The other possibility is to insert the word under the cursor. ctrl-r ctrl-w will insert the “word” under the cursor and ctrl-r ctrl-a will insert the “WORD” under the cursor (with ‘word’ and ‘WORD’ meaning what they usually do).

Here’s a random bunch of other useful expansions:

  • ctrl-r " will insert the contents of the unnamed register (i.e. the last thing you yanked or deleted without specifying a register).
  • ctrl-r + will insert the clipboard contents.
  • ctrl-r % will insert the current filename.
  • ctrl-r ctrl-p will insert the filename under the cursor, expanded in the same way as gf does. This could be particularly useful with filetype plugins that extend the behaviour of gf (like Rails.vim).

It was useful to read through the rest of the command line reference to reinforce the rest of the command line movement keys, too. So far I’d mostly just been mashing keys, assuming it behaves a bit like bash command line editing. Mostly I was right:

  • ctrl-b (not ctrl-a) to get to the start of the command line buffer.
  • ctrl-e to get to the end of the command line buffer.
  • ctrl-w to delete the word before the cursor.
  • ctrl-u to delete the characters from before the cursor to the start of the line (in other words, ctrl-e ctrl-u will delete the entire contents of the command line).
  • ctrl-c to get safely out of the command line.

Update Drew, from Vimcasts fame, has pointed out that ctrl-r works in insert mode too. That’s mind-blowingly useful. For example, you can use ctrl-r " in insert mode to paste something without exiting insert mode. I wish I’d already known that!

Symlink corruption on Mac OS X

Posted on by
Reply

Mac OS X on my desktop computer (a newish 27″ iMac, using a Promise Thunderbolt disk array for the root filesystem) seems to be having filesystem troubles. I notice it through symlinks going awry, though I’m sure they’re not the only victim. I tidied all the errant symlinks up two weeks ago, hoping it was a temporary glitch, but they’re back again today. Here’s an example:

> find -L /System -type l -print0 |xargs -0 ls -l
lrwxr-xr-x  1 root  wheel  24 15 Jan 09:42 /System/Library/Frameworks/ApplicationServices.framework/Frameworks/CoreGraphics.framework/Headers -> >File</string>????<key>L
lrwxr-xr-x  1 root  wheel  24 15 Jan 09:42 /System/Library/Frameworks/ApplicationServices.framework/Frameworks/HIServices.framework/Headers -> ?6?s?A??]h?_?:d9?r?
lrwxr-xr-x  1 root  wheel  24 15 Jan 09:42 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Headers -> >File</string>????<key>L
lrwxr-xr-x  1 root  wheel  24 15 Jan 09:42 /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Headers -> ?6?s?A??]h?_?:d9?r?
lrwxr-xr-x  1 root  wheel  24 15 Jan 09:42 /System/Library/Frameworks/ApplicationServices.framework/Versions/Current/Frameworks/CoreGraphics.framework/Headers -> >File</string>????<key>L
lrwxr-xr-x  1 root  wheel  24 15 Jan 09:42 /System/Library/Frameworks/ApplicationServices.framework/Versions/Current/Frameworks/HIServices.framework/Headers -> ?6?s?A??]h?_?:d9?r?

Each of those symlinks are pointing to some garbage. (Interestingly, the garbage quite often looks like the partial contents of a plist file.)

Here’s another example, and this is one I remember fixing last time:

lrwxr-xr-x  1 root  wheel  27 12 Nov 19:06 /System/Library/Frameworks/JavaVM.framework/Frameworks -> Versions/Current/Frameworks
lrwxr-xr-x  1 root  wheel  24 15 Jan 09:43 /System/Library/Frameworks/JavaVM.framework/Headers -> Versions/Current/Headers
lrwxr-xr-x  1 root  wheel  23 12 Nov 19:06 /System/Library/Frameworks/JavaVM.framework/JavaVM -> Versions/Current/JavaVM
lrwxr-xr-x  1 root  wheel  26 15 Jan 09:45 /System/Library/Frameworks/JavaVM.framework/Resources -> Versions/Current/Resources
lrwxr-xr-x  1 root  wheel   1  8 Jan 14:57 /System/Library/Frameworks/JavaVM.framework/Versions/Current -> c

The problem here isn’t the first four symlinks – they’re all pointing to the right places – but the last one (which they’re all pointing through) which is pointing to ‘c’, not ‘A’ like it should.

The symlink targets all seem to be the right length, just the wrong characters.

How do I go about communicating with Apple about the problem so I can get it resolved? It doesn’t really seem the sort of thing I can take to a Genius Bar…

How to install a working set of compilers on Mac OS X 10.7 (Lion)

Posted on by
Reply

With the advent of Xcode 4.2, Apple have removed GCC from the Xcode installer. Up ’til now, when you installed Xcode (say, 4.1), you’d get:

  • The original GNU Compiler Collection (GCC), version 4.2.1.

  • LLVM-GCC, which is a modified version of GCC designed to emit LLVM’s intermediate representation, so that it can hook into LLVM’s back end optimisers and code generation (as I understand it).

  • Clang, the shiny new compiler freshly built with LLVM goodness all the way through.

However, with Xcode 4.2 onwards, the original GCC variant has disappeared, and /usr/bin/gcc is now being served by the (mostly) compatible llvm-gcc.

This is only an issue if you freshly install Xcode 4.2 on a computer where the developer tools weren’t previously installed. If you upgrade from Xcode 4.1 to 4.2, it will still leave gcc-4.2 lying around (suboptimal package management, but I’m not complaining today!). I suspect this is why more people aren’t getting upset.

It’s entirely understandable for Apple to stop distributing gcc: they’re moving forward, innovating with LLVM, and there’s only so long they should have to maintain a legacy (as far as their commercial platform is concerned, at least) compiler. In fact, llvm-gcc is only there as a temporary crutch; wait ’til they remove that too, leaving us with only Clang…

However, there’s a slight snag: all compilers are not built alike. You’d think that compilers implement a standard correctly, and users of the compiler write code to that standard. It never quite works out that way, though: compilers have bugs and proprietary extensions. Worse still, coders write code until the compiler validates it, not necessarily ’til it’s right.

So, flipping to a new compiler, even gcc -> llvm-gcc, will uncover problems. One of the apps I rely on that has been having problems is Ruby, which is why I wound up messing around with this in the first place.

So, what to do? There are a couple of newly popular mechanisms out there for installing a set of compilers on Mac OS X without installing the whole Xcode behemoth (largely as a way to save disk space):

  • Soren Ionescu’s GCC Without Xcode, which uses the Xcode installer you download from the App Store to generate a slimline installer with just the bits you need. Soren is careful to do it this way so as not to distribute any of Apple’s binary packages on their behalf (something they, naturally, can get a little grumpy about). Unfortunately, this means it’s using the Xcode 4.2 installer, which … doesn’t have GCC.

  • Kenneth Reitz’s OSX GCC Installer, which, in the project’s download section, has a pre-built package, ready for installation. Here’s the key thing: the pre-built package is built against Xcode 4.1, so still includes gcc-4.2.1. Hallelujah, we’re saved. (Kenneth, please for the love of all things that compile, please don’t update that package!)

So, when you’ve got a fresh Lion installation, and you’re looking for something you can build the widest range of apps on (from things that require gcc all the way through to that shiny iOS 5 project you’re working on), do the following:

  • Grab the OSX GCC Installer (as of writing, the 10.7-v2 package is the way forward).

  • Run the package to install it.

  • Grab Xcode from the App Store.

  • Run the Xcode installer.

Finally, you should be good to go. /usrbin/gcc will still point to llvm-gcc-4.2 but at least you’ll have gcc-4.2 installed. If you’re having trouble compiling things, try forcing it to use gcc with CC=gcc-4.2 or equivalent. I think the likes of Homebrew and RVM will try to be clever on your behalf if they can.

Running tmux in Mac OS X Terminal

Posted on by

I’ve been a fan of screen for … a while now. But since I like being one of the cool kids, I’ve been using tmux for the past year or so. Last week, I noticed that every time I launch a new terminal, I wind up typing tmux attach-session. Let’s streamline, a little bit.

In Mac OS X’s Terminal.app, you can change the shell that it runs. Here’s how I did it:

  • Open Preferences, and choose the Settings tab.

  • Duplicate your existing settings (since sometimes you might not want tmux after all). Pick your default session (mine’s “Pro”) and select “Duplicate settings” from the tool menu at the bottom. Name the new settings “Tmux” or something along those lines.

  • In the shell tab for your settings, select “Run command” and enter /usr/local/bin/tmux attach-session. Deselect “Run inside shell” since you don’t really need to. Since you’re not running inside a shell, /usr/local/bin probably isn’t in your $PATH so you’ll need to specify the full path name. Of course, if your tmux binary lives somewhere other than /usr/local/bin you’ll need to change the path.

  • If you’ve selected “Only if there are processes other than” for “Prompt before closing”, then you’ll probably want to add tmux to that list.

  • In the “Window” tab, I set “Scrollback” to limit the number of rows to ’0′, since tmux provides scroll back, and the Terminal one isn’t terribly useful when tmux is running inside it.

  • Make sure your Tmux session is set as the default one by clicking the “Default” button at the bottom of the settings lists while it’s selected.

That’s it. Close your existing terminal sessions and launch a new one. You should be launched into (one of) your existing tmux sessions. If tmux wasn’t already running, then this assumes that your ~/.tmux.conf sets up at least one session (which I think it required anyway). If you’ve got more than one tmux session running, I’ve no idea which one, offhand, it’ll choose, but you can always switch to the one you’re looking for with C-a s. (You have rebound the prefix to C-a, right?)

There’s (at least) one time where you don’t want tmux as your shell. That’s when you’re attempting to interact with launchd. I suspect it’s to do with launchd checking that you’re a child process as part of its permissions when you’re asking it to do stuff, where tmux works by detaching itself from its parent process so it’s not killed when the parent is. (Total guess, BTW.) Still, when you want to use launchctl, you’ll need to do it somewhere other than a tmux session. In Terminal, choose the Shell menu, choose “New Window” (or “New Tab”) and select one of the other settings profiles.

Give me back my # key!

Posted on by

This particular tip may have an audience of approximately one, since:

  • It’s only going to bother Mac users;
  • who are in the UK;
  • who use the # (hash, pound, whatever you call it) key much; and
  • who have enabled “Use option as meta key” in Terminal.app in order to sanely use Emacs.

If you tick all these boxes, and have suddenly discovered that your # key (which is option-3 for UK-keyboard-wielding Mac users) no longer works, you’ve come to the right place!

The problem is that M-3 is bound to digit-argument. This allows you to repeat commands (e.g. if you type M-3 c, then it will output ccc) which I’ve got to admit I don’t use terribly often.

It turns out that the bindings for keys are controlled by the readline library and you can customise them with readline’s configuration file, ~/.inputrc. If you want to override the default behaviour of M-3 and turn it back to emitting the # symbol, put the following in that file:

"\e3": '#'

You’ll need to restart bash (or force it to reload its inputrc file with C-x C-r) in order for it to take effect.

Here’s another related tip, while I’m here. I picked this one up from Ross a few years ago. I almost never use the UK currency symbol (£) in a shell environment. On the other hand, there’s another operation I perform quite often at the command line: commenting out the command I’m currently typing. It usually happens when I’m doing a sequence of steps at the command line, and I realise that I’ve forgotten a prerequisite step. Rather than sticking the current line in the kill ring (C-e C-u or C-a C-k), then remembering to yank it (C-y) again, I tend to jump to the start of the line (C-a), stick a hash in (to comment the line out), then hit enter. That way it’s in my (searchable) bash history for when I next need it.

But that’s a fairly cumbersome sequence too, so let’s shorten it. Stick the following in ~/.inputrc:

"£": '\C-a#\C-m'

and restart your shell. Now when you’re part way through typing a command line and want to switch tracks, hit the £ sign and you’re done. When you want the command back, retrieve it from your shell history (C-r to search!), then hit C-a C-d to remove the comment sign and you’re good to go.

One last wee trick. You really can remap any key to any other key. Imagine the fun and hilarity of the following in your ~/.inputrc:

"l": 'r'
"s": 'm'

:-)

Update: My colleague, Mihai, points out that the only appreciable difference between the UK and US keyboard layout on the Mac is that the # and £ key combinations are swapped around (oh, and it’s easier to find the ™ than the € now, not that I use either). Since I’ve just discovered that my irb isn’t picking up the ~/.inputrc and doing the right thing, I reckon I’ll just switch to the US layout…

Understanding the Rails Logger

Posted on by

I post here fairly infrequently and irregularly. I’m sure the fact that I’m posting over at the FreeAgent Engineering Blog too isn’t going to help that at all.

Just yesterday I had a bit of an adventure in the Ruby on Rails logging code. You can find the article on the engineering blog: Understanding the Rails Logger.

If Ruby, Rails, testing, scalability, automating your infrastructure and a wee smattering of accounting is the kind of thing you’re interested in, you could do worse than keeping track of FreeAgent’s engineering blog. I’ve got some very smart colleagues with some really interesting stuff to say…

Ruby Timeout Woes, Part 2

Posted on by

I started digging into how Ruby’s timeout mechanism worked this morning, in order to get to the bottom of a bug we’ve got.

Let me give you a little context. We use Delayed Job to run some of our longer running tasks. Delayed job wraps all its jobs in a timeout, which we’ve set to 20 minutes. That’s a good thing: I don’t really want a job running forever and, consequently, tying up one of our workers forever. So, we’ve got Delayed Job wrapping arbitrary code in Ruby’s built in Timeout. What can possibly go wrong?

Well, it turns out that, for one particular job, the timeout mechanism wasn’t working, and the job was carrying on well past the 20 minute timeout we’d set. Worse still, when a running job exceeds the maximum run time, Delayed Job will assume that the entire worker died, break the lock and hand the job to another worker. So we wound up with every single delayed job worker in our cluster running the same job, to completion, no matter how long it took.

Suboptimal, eh?

I started digging into Delayed Job, our code, and the Timeout implementation to see if I could figure out what was going wrong. Delayed Job is doing fine, nothing unusual there. The Timeout implementation is interesting. It creates a separate thread, which then sleeps for the timeout length. If the main thread completes its block before the timeout, it just kills the timeout thread and carries on happily. However, if the timeout thread wakes up before the main thread has completed execution, then it raises an exception on the main thread. The timeout method catches that exception on the main thread, tidies up and raises a Timeout::Error exception.

There are a few problems with that implementation (every call to Timeout.timeout creates a new thread, and it makes use of Thread.raise and Thread.kill which, as Charles Nutter pointed out a few years back is a little broken), but we’ll gloss over them for now. That’s not what was causing my woes today. Let’s reduce the problem to a simple example:

require 'timeout'
 
puts "#{Time.now}: Starting"
begin
  Timeout.timeout(5) do
    begin
      sleep 10
    rescue Exception => e
      puts "#{Time.now}: Caught an exception: #{e.inspect}"
    end
    sleep 10
  end
rescue Timeout::Error => e
  puts "#{Time.now}: Timeout: #{e}"
else
  puts "#{Time.now}: Never timed out."
end

Let’s see what happens when we run that wee snippet:

Tue Aug 30 13:38:56 +0100 2011: Starting
Tue Aug 30 13:39:01 +0100 2011: Caught an exception: #<#<Class:0x1001337f0>: execution expired>
Tue Aug 30 13:39:11 +0100 2011: Never timed out.

The inside rescue block is catching some exception after the timeout has expired, but the one expecting the timeout error never gets it. That’s down to the implementation of Timeout. When the timer thread reawakened, it threw an exception on the main thread. The exception it threw on the main thread inherits from Exception, so anything that catches Exception will catch it before it bubbles back up the stack to the timeout method. So, while we’ve timed out the inner block, we’ve neutered the overall effect of the timeout method.

Lessons learned:

  • Catching generic StandardError exceptions is crazy enough, but you probably never want to catch Exception. PS, library authors, your exceptions should inherit from StandardError, not Exception.

  • Ruby’s built in Timeout mechanism is crazy in a whole new and interesting way, too. Be careful how you use it.