Edge Rails (pre-2.2), iconv, transliteration and Solaris
Thursday, October 16, 2008 at 4:20PM It would appear that the C library implementation of `iconv(3)` on my build of OpenSolaris (Nevada build 99) doesn't actually support transliteration, so this code was raising an exception:
Iconv::InvalidEncoding: invalid encoding ("ascii//translit//IGNORE", "utf-8")
every time the inflector was required (which was every time Rails was loaded in any way). That sucked somewhat because it meant I couldn't deploy an application I was working on!
It took me a while to realise that the C library implementation of iconv here was somewhat lacking (I assumed everybody just used GNU iconv these days). Turns out the solution is to install the SUNWgnu-libiconv package on Nevada:
bash-3.2$ pfexec pkg install SUNWgnu-libiconv
That'll install the appropriate libraries but, as you can see, all is still not well:
bash-3.2$ irb
irb(main):001:0> require 'iconv'
=> true
irb(main):004:0> Iconv.iconv('ascii//translit//IGNORE', 'utf-8', "fôôø").to_s
Iconv::InvalidEncoding: invalid encoding ("ascii//translit//IGNORE", "utf-8")
from (irb):4:in `iconv'
from (irb):4
from :0
You also must force the GNU version of the library to preload so that it takes over from the built-in implementation of `iconv(3)`:
bash-3.2$ LD_PRELOAD="/usr/gnu/lib/preloadable_libiconv.so" irb
irb(main):001:0> require 'iconv'
=> true
irb(main):004:0> Iconv.iconv('ascii//translit//IGNORE', 'utf-8', "fôôø").to_s
=> "f^oo^o"
That's all well and good for individual sessions of `irb(1)` but that's going to be somewhat cumbersome to roll into all of our Rails deployments. I'm still trying to find a workable solution to apply this "globally". Here's what I've tried so far, and failed to make work:
* Find a sensible global place to set LD_PRELOAD so that every application will catch it. This seems somewhat scattergun and a bit dangerous.
* Replace the ruby binary with a shell script that sets LD_PRELOAD then calls the real Ruby binary. This sorta worked, but it breaks every she-bang-ed script because `#!/path/to/ruby` is no longer an ELF binary, but a shell script. Chances are this would be a winning strategy if I could figure out how to code it in C.
* Relinking the `iconv.so` Ruby library to add a reference to `libgnuiconv.so`. This was a total stab in the dark because I don't even know if you can relink dynamic libraries after the fact.
Needless to say, none of these strategies worked, so I think I'm back to recompiling Ruby from source. Yee ha. Perhaps now is a good time to try out [Ruby Enterprise Edition](http://www.rubyenterpriseedition.com/), see what happens...
Ruby and Rails,
Solaris,
Work
Reader Comments (4)
If you want to change this globally you can also toy around with the crle command to change the default search path that ld.so uses. Not that I'd recommend it in this case, but still its an option for this type of issue.
Your best approach is to rebuild ruby to point at a different version of libiconv.
@Andy I think it's insane to change the default behaviour of
iconv(3)globally for the entire system to use GNU iconv. But on the other hand, I might actually be insane. So far it's working! The magic invocation to make it happen is:Thanks for the pointer to
crle. :-)Seems as if the Coolstack offering from Sun (http://cooltools.sunsource.net/coolstack/) on vanilla x86 Solaris 10 (SunOS xxx 5.10 Generic_127112-11 i86pc i386 i86pc) suffers from a similar iconv problem.
There exists a file: /opt/csw/lib/libiconv.so.2.2.0, so I don't know if some sort of preload/symlink voodoo will convince it to work...
How have you got on with Enterprise Edition?
@Graeme If you use crle to globally preload GNU iconv then you may also need to set the /usr/gnu/lib directory as secure, via this command:
crle -s /lib/secure -s /usr/lib/secure -s /usr/gnu/libOtherwise, you may get errors such as:
ld.so.1: pfexec: warning: /usr/gnu/lib/preloadable_libiconv.so: open failed: illegal insecure pathname