Backstory

Some days ago, at work I had to do regular expression matching with some UTF-8 or Unicode^[1] encoded text. That made me read perlunitut. After reading it, I got to know that UTF-8 & Unicode are not the same^[2]. There’re many Unicode encodings & UTF-8 is the most commonly used Unicode encoding.

Today, once again I was bitten by my code.This is what I did there (in brief):

• I store details of an event in a YAML file.
• Construct an HAML file using data from the YAML file.
• Construct a .tt (Template Toolkit) from a HAML file.
• Finally, an .html file using that .tt file.
• Also, the unicode character appeared malformed in the email.

In YAML file, one of the speakers has a unicode character in his name, which made it appear on the browser like this : When I tried to print the variable having UTF-8 text, I got this warning :

Wide character in print

Solutions

In my code, I was using quite a few modules namely :

• IO::All for writing text to .haml file.
• Text::Haml for rendering text of .haml file.
• Template for rendering .tt file.
• Email::Simple for sending emails.

IO:All

I was using IO::All::io for reading & writing files. You can enable UTF-8 encoding in IO::All, by two ways (from the man page of IO::All):

• Enable UTF-8 for a single operation:

$contents = io('file.txt')->utf8->all;  # Turn on utf8

• Enable UTF-8 for all operations:

use IO::All -utf8;                      # Turn on utf8 for all io
$contents = io('file.txt')->all;        # by default in this package.

Text::Haml

In Text::Haml, you can set text encoding while creating an object of Text::Haml, like this :

my $haml   = Text::Haml->new( encoding => 'utf-8' );

Template

In Template, you can specify encoding as an argument to process, like this:

$tt->process(
    'input.tt',
    { key => "value"},
    'output.html',
    binmode => ':utf8'  # set encoding to UTF-8
)

Email::Simple

In Email::Simple, you can specify the encoding in the email headers, like this:

my $email = Email::Simple->create(
    header => [
        To                   => $to,
        From               => $from,
        Subject           => $subject,
        'Content-Type' => "text/plain; charset=utf-8",
    ],
    body   => $email_body,
);

print statements

For getting rid of this warning:

Wide character in print

You can do this:

# This allows you to write in UTF-8 in your program
use utf8;

# This tells STDOUT that it should see everything as UTF-8, instead of bytes
binmode STDOUT, ':utf8';

Better solution?

The bad part of the solutions is that I had to enable UTF-8 for every module separately. I want to know a way to enable UTF-8 through out the script for everything. I’ve not found any way to do this :(. So, if you know about any such way, then please let me know :).

^[1]Don’t dare stop reading here, if you will do then you may live in the dark.
^[2]Reason behind ^[1]