Search

Enter a search word or two and press return to see the search results.

Who am I?

Hi, I’m Graeme and these are my notes, from my messy desk. I started this blog because Google proved to be more useful at finding content than anything else I’ve used.

So I started adding my own content in the hopes that Google would index it and allow me to find things again in the future.

It works.

You can find out more about me here, and you should follow me on Twitter here.

Keeping up

You can automatically receive new content here by subscribing to the “Blog RSS” (link below). This is the easiest way to keep up with what I write here.  See this BBC article for a good introduction on RSS and keeping up with the goings on of the Internet more easily.

« Hobbling the iPhone | Main | First shot with my new Canon EOS 50D! »
Thursday
16Oct2008

Edge Rails (pre-2.2), iconv, transliteration and Solaris

I've been having trouble with edge Rails on Solaris for the past few days, and I finally decided to figure out what the problem actually was. It all boiled down to this method here (from [activesupport/lib/active_support/inflector.rb](http://github.com/rails/rails/tree/e0993c6c376d62716757a8a7f476ed0c369d0fc7/activesupport/lib/active_support/inflector.rb)#275):



It would appear that the C library implementation of `iconv(3)` on my build of OpenSolaris (Nevada build 99) doesn't actually support transliteration, so this code was raising an exception:

Iconv::InvalidEncoding: invalid encoding ("ascii//translit//IGNORE", "utf-8")

every time the inflector was required (which was every time Rails was loaded in any way). That sucked somewhat because it meant I couldn't deploy an application I was working on!

It took me a while to realise that the C library implementation of iconv here was somewhat lacking (I assumed everybody just used GNU iconv these days). Turns out the solution is to install the SUNWgnu-libiconv package on Nevada:

bash-3.2$ pfexec pkg install SUNWgnu-libiconv

That'll install the appropriate libraries but, as you can see, all is still not well:

bash-3.2$ irb
irb(main):001:0> require 'iconv'
=> true
irb(main):004:0> Iconv.iconv('ascii//translit//IGNORE', 'utf-8', "fôôø").to_s
Iconv::InvalidEncoding: invalid encoding ("ascii//translit//IGNORE", "utf-8")
from (irb):4:in `iconv'
from (irb):4
from :0

You also must force the GNU version of the library to preload so that it takes over from the built-in implementation of `iconv(3)`:

bash-3.2$ LD_PRELOAD="/usr/gnu/lib/preloadable_libiconv.so" irb
irb(main):001:0> require 'iconv'
=> true
irb(main):004:0> Iconv.iconv('ascii//translit//IGNORE', 'utf-8', "fôôø").to_s
=> "f^oo^o"

That's all well and good for individual sessions of `irb(1)` but that's going to be somewhat cumbersome to roll into all of our Rails deployments. I'm still trying to find a workable solution to apply this "globally". Here's what I've tried so far, and failed to make work:

* Find a sensible global place to set LD_PRELOAD so that every application will catch it. This seems somewhat scattergun and a bit dangerous.

* Replace the ruby binary with a shell script that sets LD_PRELOAD then calls the real Ruby binary. This sorta worked, but it breaks every she-bang-ed script because `#!/path/to/ruby` is no longer an ELF binary, but a shell script. Chances are this would be a winning strategy if I could figure out how to code it in C.

* Relinking the `iconv.so` Ruby library to add a reference to `libgnuiconv.so`. This was a total stab in the dark because I don't even know if you can relink dynamic libraries after the fact.

Needless to say, none of these strategies worked, so I think I'm back to recompiling Ruby from source. Yee ha. Perhaps now is a good time to try out [Ruby Enterprise Edition](http://www.rubyenterpriseedition.com/), see what happens...

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (4)

If you want to change this globally you can also toy around with the crle command to change the default search path that ld.so uses. Not that I'd recommend it in this case, but still its an option for this type of issue.

Your best approach is to rebuild ruby to point at a different version of libiconv.

October 16, 2008 | Unregistered CommenterAndy Steingruebl

@Andy I think it's insane to change the default behaviour of iconv(3) globally for the entire system to use GNU iconv. But on the other hand, I might actually be insane. So far it's working! The magic invocation to make it happen is:

bash-3.2$ pfexec crle -u -E LD_PRELOAD=/usr/gnu/lib/preloadable_libiconv.so

Thanks for the pointer to crle. :-)

October 16, 2008 | Unregistered CommenterGraeme Mathieson

Seems as if the Coolstack offering from Sun (http://cooltools.sunsource.net/coolstack/) on vanilla x86 Solaris 10 (SunOS xxx 5.10 Generic_127112-11 i86pc i386 i86pc) suffers from a similar iconv problem.

/opt/coolstack/lib/ruby/1.8/i386-solaris2.10/iconv.so: ld.so.1: ruby: fatal: libiconv.so.2: open failed: No such file or directory - /opt/coolstack/lib/ruby/1.8/i386-solaris2.10/iconv.so (LoadError)
from /opt/coolstack/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /export/home/sam/src/infra/lumberyard/vendor/rails/activesupport/lib/active_support/inflector.rb:3
...

There exists a file: /opt/csw/lib/libiconv.so.2.2.0, so I don't know if some sort of preload/symlink voodoo will convince it to work...

How have you got on with Enterprise Edition?

January 2, 2009 | Unregistered CommenterSam Pointer

@Graeme If you use crle to globally preload GNU iconv then you may also need to set the /usr/gnu/lib directory as secure, via this command:

crle -s /lib/secure -s /usr/lib/secure -s /usr/gnu/lib

Otherwise, you may get errors such as:

ld.so.1: pfexec: warning: /usr/gnu/lib/preloadable_libiconv.so: open failed: illegal insecure pathname

January 20, 2009 | Unregistered CommenterJ. Aaron Farr

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>