Recently in Perl programming Category

The FFmpeg::Command Perl module is a convenient way to drive the ffmpeg command-line utility for converting multimedia files.

For work, I have developed some scripts that make heavy use of FFmpeg::Command. Yesterday, one of the other developers told me they need a conversion script to be able to merge separate video and audio streams into one file that contains both audio and video. The ffmpeg command-line utility can do this by accepting more than one input file. For example:

$ ffmpeg -i video.avi -i audio.wav -acodec copy -vcodec copy merged.avi

The FFmpeg::Command Perl module, however, assumed there can only be one input file. I made the necessary changes to the module code so that it would accept multiple input files, created a patch file, and sent it to the Module owner Gosuke Miyashita. This morning, I received e-mail from Gosuke thanking me for the patch and informing me that he has uploaded a new version (v0.12) of FFmpeg::Command to CPAN.

I love Perl and open source software!

I think I have finally, really arrived.

I’ve been doing contract work for a company in Provo that is launching a new website called YoManSports.com, which is in beta right now. At first glance, the site may appear to be a “YouTube for sports,” but it so much more than that. The concept is centered around video sharing, but includes familiar social networking elements you’d find on sites like Facebook or MySpace. In addition, there are several applications within the site that are sports-related — things like competition bracketing, scorecards, and groups. Perhaps the coolest feature that rounds out the list is the broadcast feature. This lets a person go to a sporting event with a video camera, even something as simple as a USB webcam, and set up a live web broadcast that anyone with a web browser can watch. The person managing the broadcast can mix prerecorded video, pictures, and even live video from other users into the broadcast. There’s even a news ticker for embedding clickable URL links into the broadcast. It’s pretty cool stuff.

Now, I said at the beginning of this post that I have arrived because we’ve been asked by management to blog regularly about the site and what we’re doing with it as part of our marketing plan. So, yeah, it’s cool to be able to do this and not be wasting my time at work.

My job has been designing and building the server architecture that sits behind the scenes and makes it all work. I was brought in late 2008 when the site was pretty much in a prototype stage. All the code was running on a single server and it really wasn’t designed to scale beyond that one server. So, one of the first things I did was figured out what we’d need to do split things like streaming video, web services, and database services onto their own dedicated servers.

After that, I went through and figured out how we were going to accomodate loads higher than we could with individual servers. In a nutshell: load balancing. That has now been implemented.

Another thing I’ve had a big hand in is offloaded encoding and conversion. The developers had created routines to do all the video encoding in PHP on the frontend of the website. Of course, doing video encoding on the same server Apache is running on can be detrimental to the experience of other website users. I developed a distributed encoding system that handles all the video conversion and encoding on a separate set of servers. I did it with Perl, of course.

I’m pleased with the technology being used on the site. I’m not a fan of PHP, but it’s doing the job well for frontend development. We’re making use of a lot of open source technology in dealing with videos. For example, all our transcoding is being done with the formidable FFMPEG software along with libraries like x264 and FAAC.

We’re leveraging Flash pretty heavily pretty heavily to make the site work so it’s fortunate that Flash support has nearly ceased being a problem for cross-platform compatibility. YoManSports.com works almost seamlessly across Windows, Mac OS, and Linux.

Watch this space for more info to come.

Hey y’all, I’ve volunteered to teach in the Fedora Classroom this Saturday (7 Mar 2009). The Fedora Classroom is an IRC-based classroom environment.

So, at 3pm MST (22:00 UTC), anyone can participate by logging in to #fedora-classroom on irc.freenode.net and I, fozzmoo, will be doing a 1-hour presentation on Perl basics.

I’ve been digging through old presentations and workshops notes from when I used to do all day Perl workshops at USU for the USU Free Software and Linux Club to see what I can distill down into a 1-hour presentation. If there’s enough interest and response, we’ll see about turning this into a regular thing.

(Ryan Byrd)[http://www.ryanbyrd.net/techramble/] blogged recently with a (programming interview question)[http://www.ryanbyrd.net/techramble/2009/03/03/programming-interview-question-of-the-day/] that I thought I’d take a stab at in Perl.

The question is as follows:

• when passed in a number that is evenly divisible by 3, return “wiz”
• when passed in a number that is evenly divisible by 5, return “bang”
• when passed in a number that is evenly divisible by both 3 and 5, return “wiz bang”
• otherwise, return the number passed in

My solution exploits Perl’s list type to store potential output as a queue of sorts.

sub function {
    my $num = shift;
    my @output = ();
    unless($num % 3) {    push @output, "wiz"; }
    unless($num % 5 ) {    push @output, "bang"; }
    if(@output) {   return join ' ', @output; }
    return $num; 
}

Working with databases is something programmers, especially Web programmers, often need to do. Most (reputable) database backends provide a way to use Structured Query Language (SQL) queries to interact with the database. That’s usually where the similarity ends. Working with MS SQL Server, Oracle, and MySQL databases typically means you must acquire connection libraries unique to a specific database backend to interact with a database with SQL.

Since we’re talking about Perl, let’s use PostgreSQL as an example. There is a CPAN module called Pg that gives you a set of subroutines for interacting directly with a PostgreSQL database backend.

Here is the example usage from the Pg POD:

use Pg;
my $conn = Pg::connectdb("dbname=template1");
my $res  = $conn->exec("SELECT * from pg_user");
while (@row = $res->fetchrow) {
    print join(" ", @row);
}

If you wrote a whole application using this Pg module and then someone came along and said, “Hey, I like your application, but we use MySQL,” you’d probably plant your face into your palm pretty hard.

DBI

In the 1990s, Tim Bunce contributed the DBI module to CPAN. DBI is database abstraction layer meaning that it sits between your applications and any database backend and gives you (the programmer) a generic set of facilities for interacting with databases.

Examine at how the Pg example could be accomplished using DBI:

use DBI;
my $dbh = DBI->connect('dbi:Pg:dbname=template');
my $sth = $dbh->prepare('SELECT * FROM pg_user');
my $rv = $sth->execute;
while(my @row = $sth->fetchrow_array) {
    print join(' ', @row;
}

The first thing to notice here is the DBI->connect() line. In many cases, this line is the only one you would need to change to migrate a DBI application from one database backend to another. The first parameter passed to the connect() function is a DBI Data Source Name — or DSN.

Some examples of DSNs that you may use with DBI:

• dbi:mysql:database=shoppinglist;host=db1
• dbi:Pg:dbname=bookshelf;host=192.168.1.22;port=5432
• dbi:SQLite:dbname=/var/db/addrbook.db
• dbi:CSV:f_dir=/home/joe/csvdb
• dbi:Oracle:host=oracle;sid=oracle

DBD modules

The interface from DBI to each specific database backend is provided by DBI Drivers or DBD modules. In addition to drivers for most common database backends, there are some unusual and unique drivers as well such as DBD::CSV which provides the means to use SQL queries to interact with data in comma-separated values text files.

Stop worrying about quoting

One thing that typically comes up when working with non-DBI database interaction methods is worrying about value quoting. SQL requires that column values be quoted with single-quote characters unless the value is a number. For example:

INSERT INTO TABLE friend (first_name, last_name, age) 
VALUES ('Joe', 'Smith', 15);

This is, of course, assuming the age column is an integer. Think about zip codes. The person who designed the schema for the database you’re working with might have made the assumption that a zip code would always be a 5-digit number and therefore defined the zipcode as an integer type. Another person might have considered the possibility of zip+5 zip codes and defined the column as VARCHAR(10) and values would therefore need to be quoted inside single quotes.

Fortunately, if you use DBI properly, you won’t have to worry about quoting because you can use placeholders and bind values in query strings. See the example below:

my $sth = $dbh->prepare('INSERT INTO friend (first_name, last_name, age) 
VALUES (?, ?, ?)');
my $rv = $sth->execute( $first_name, $last_name, $age);

The question marks in the prepare() call are placeholders and the parameters passed to execute() are the corresponding bind values.

One of the reasons DBI uses a prepare() call followed by an execute() call instead of one call to execute a query is so you can reuse a prepared query with multiple bind values. Notice this example which reads from a CSV file and populates a database table:

my $sth = $dbh->prepare('INSERT INTO friend (first_name, last_name, age) 
VALUES (?, ?, ?)');

while(<CSV>) {
    my ($first_name, $last_name, $age) = split /,/;
    my $rv = $sth->execute( $first_name, $last_name, $age);
}

Fetching data

In the first example above which showed how the Pg module interacted with a PostgreSQL database backend, the fetchrow() call returned an array of values in a row of results. This is fairly limited and by no means provides the result data in all the ways a programmer would to use it. For example, one glaringly absent piece of information is the field names.

DBI provides multiple calls for fetching result data. Below is an example of the fetchall_hashref() call which gives you access to all rows in a result as a referenced hash.

my $sth = $dbh->prepare('SELECT * FROM friends');
my $rv = $sth->execute;
my $friend_hash = $sth->fetchall_hashref('id');

For simple queries like this, it may makes sense to use DBI’s selectall_hashref() call, which results in even fewer lines of code:

my $friend_hash = $dbh->selectall_hashref(
    'SELECT * FROM friends', 'id');

The resulting hashref, when dumped using Data::Dumper might look like this:

$VAR1 = {
      '1' => {
               'id' => 1,
               'age' => 15,
               'last_name' => 'Smith',
               'first_name' => 'Joe'
             },
      '3' => {
               'id' => 3,
               'age' => 33,
               'last_name' => 'Jansen',
               'first_name' => 'Stuart'
             },
      '2' => {
               'id' => 2,
               'age' => 25,
               'last_name' => 'Johnson',
               'first_name' => 'Roger'
             }
    };

In conclusion

This short article only scratches the surface on DBI, but it hopefully gives the reader an idea of the power and flexibility provided by this valuable CPAN module.

For those who are looking for more indepth information O’Reilly and Associates has published a fine book on DBI, co-written by Tim Bunce, Programming the Perl DBI which is highly recommended. And then there’s always the DBI PODs: Type perldoc DBI after/if you’ve got DBI installed on your system.

I’ve had an analogy on my mind for several days, churning, and I think I’d better blog it or else it’s going to devour me. Here it is:

“PHP application development is like bologna as Perl application development is like a good marinated London Broil.”

So, there you go. Now, let me explain. There are upsides and downsides to this.

Balogna requires nearly nothing to start eating. It’s as simple as opening the package, removing a slice of the processed meat product, and eating it. It’s just that easy. My five-year old does it all the time. Sometimes, he puts it on a plate and, using a butter knife, cuts the slice of balogna into little wedge-shaped pieces.

And, as a consumer, you don’t have to do much to get balogna to a point where you can consume it. It’s ready in its edible form at the supermarket. You go, buy it, take it home, (cut a slice into little wedge-shaped pieces), and eat it.

London Broil, on the other hand, isn’t available at the supermarket in sealed ready-to-eat packaging. You have to find a cut of top round beef, preferably one that has a nice amount of thickness (over 1-inch thick is ideal). In addition to the beef, you’re going to need to get some other ingredients for the marinade. When I was in college and wanted to impress a girl on a date, my sister suggested preparing a dinner that included London Broil. I don’t remember the exact recipe for the marinade now, but I remember there was worchestershire sauce, red cooking wine and series of different spices and salts. Looking online, there are many different recipes for such a marinade that include honey, garlic, chopped parsley, pepper, soy sauce, and more.

Once you get your ingredients home, mix the marinade and put it in a large plastic bag that zips shut. Place the raw beef into the bag with the marinade and put it in the refrigerator for up to 24 hours, turning it over once.

Cooking the marinated London Broil is a tricky p;ocess. You definitely broil or grill the meat, but this isn’t a hamburger. You need to heat it carefully, about six to eight inches over the flames. Some recipes actually call for broiling the meat to “rare” before you place it in the marinade. After the meat is cooked to the desired wellness, take it off the grill and begin cutting it in slices with your knife at a 45-degree angle to the meat.

Mmmmm mmmm. This is making my mouth water, just blogging about it!

London Broil, a magnificent meal that is sure to impress a date (or scare her away, as was my case.).

To bring this back to the programming languages, let me regurgitate a story of a recent experience I had.

A couple of weeks ago, a handful of systems I used to manage were compromised. These servers were running Fedora Core 6 and Fedora Core 5 which, of course, haven’t been supported by the Fedora community in at least a couple of years. The obvious response to a compromise was to install a actively supported Linux distribution on the hardware, make sure security issues are addressed, and then re-deploy the applications.

I was asked to assist in the process because of my knowledge of the systems.

Backing up critical data, installing a new OS, restoring data was pretty straightforward and easy. Next came the harder part: Getting all the applications configured and working again, just as they were supposed to.

Most of these applications were written in Perl. I went through each, one by one, taking note of the errors that occured when I first tried to run them. These errors invariably complained of missing CPAN modules — most of which were not available as packages available to install from a software repository affiliated with the Linux distribution.

This meant I had to go through, one by one, and build packages for each CPAN module that was a dependency for the applications. Many of these had their own sets of dependencies. The result: a couple of hours building packages to satisfy an interconnecting web of dependencies. In the end, everything worked.

It was then I decided to check on the one application that wasn’t written in Perl. This last application was written in PHP by some no-name programming team that the client paid to develop and host the software but when they didn’t have the chops to host and manage the application, the client took the application to someone who did have the necessary skills.

Guess what. The PHP application ran, out of the box. no unmet dependencies; No package installations neededl No fuss; Nothing.

At a recent PLUG meeting, I shared my experience with a friend who nodded in agreement. He too had his share of trouble navigating the waters of “dependency hell” trying to get a Perl application working.

Some Perl developers don’t understand this experience because they don’t use their OS’s packaging infrastructure to manage their Perl installation. Instead, they let Perl run loose, so to speak, and install necessary packages outside of a package management system. The upside: It’s fast. The downside: It’s risky and unmanagable.

My friend said, “Perl developers are just too smart.” I have to agree. Damian Conway, a Perl guru that travelled to Utah and spoke to a group of us about four years ago, likes to quote the late Arthur C. Clarke: “Any sufficiently advanced technology is indistinguishable from magic.” Conway prides himself on being the author of several CPAN modules that work “like magic.”

Conway’s not the only one. Many in the Perl community have developed extremely useful but complicated pieces of code and have graciously shared it with the open source community. Like any good open source community, others have built on what has been done and the result is software that has a deep root system of module dependencies.

Meanwhile, Perl has fallen out of favor as a language of choice for web application development, despite all the “magic” that exists within the Perl community. Why? If it’s so technologically superior, why aren’t the hordes of web developers using it?

The answer: Precisely because it is so technologically superior.

Newcomers to web development are drawn to the simplicity, straightforwardness, and relatively painless entry of developing applications using PHP. I can’t tell you how many PHP-based web applications I’ve looked at that are quite useful and powerful on the outside, but the code is... well, it’s boring. That’s not to say the programmers didn’t know what they were doing, they just didn’t really seem to know of any optimal ways of doing it. In the end, that’s okay, because the software works as it’s supposed to. A computer scientist will tell you, however, that this type of programming runs the risk of becoming unmanagable as it grows. The PHP community doesn’t seem to mind, though. They’ve had no problem “brute-forcing“ their way through most obstacles like this.

As a advocate of Perl, I’m left with a problem. My language of choice is failing at popularity contests and now I think I know why: It’s a pain in the ass to grasp in order to wield the magic.

What is it going to take to rectify this problem?! A Linux distribution that includes a great portion of the CPAN modules a programmer would ever need? That would certainly make things a lot easier.

What if the great minds of Perl put their current challenges aside for a few moments and tackled this challenge instead? Make the magic easier to obtain. Make using Perl much less frustrating for the uninitiated.

There are some that would say selling London Broil in a ready-to-eat package would be too hard or that it would be too expensive. I don’t know. I know you can get some very tasty prepared, marinated meats ready to slap on a plate. Sure, you pay more than you would for a slice of balogna, but isn’ it worth it?

I’m still a little miffed I can’t sync my Centro with my KDE PIM suite, but I did just today find a reason to increase my love for the Fedora Project.

With previous versions of Fedora (and RHEL/CentOS), I’ve had to manually build some Perl modules like DBIx::Class and Catalyst. Lo and behold, there exists a handy-dandy perl-DBIx-Class package in the standard Fedora repositories.

What a complete and awesome joy it was to watch perl-DBIx-Class download and install along with the 45 or so dependencies it required. It’s so nice to not have to build that manually with cpan2rpm anymore!

This weekend, I installed Scalix on a client's mail server. I had installed Scalix before, but it was a clean install with no old e-mail to migrate. This time, however, I had a lot of old e-mail to migrate.

Most of the users used Eudora for Windows. Others used the openwebmail web mail client. As a result, I had e-mail from three different sources to populate with:

• Eudora locally-saved mailboxes (an almost mbox-format)
• IMAP folders on the server, in mbox format
• Standard mbox mailspool boxes (inboxes).

In preparation, I had read on a page on the Scalix site there were scripts to import mbox files into Scalix, but as it turned out, these scripts were for restoring a proprietary-to-Scalix backup mailbox format. Bleh.

The next best solution was to use an e-mail client that could import mbox mailboxes and move the messages to an IMAP folder. Thunderbird only imports a few formats on Windows and only imports Netscape Communicator on Linux. Bleh.

So, I tried kmail. It worked beautifully. At one point, I had three instances of kmail running in three different VNC sessions on the server funneling messages into Scalix. The only problem with kmail is that it required constant babysitting as I moved from mailbox to mailbox and user to user.

So, while kmail was cranking along on a large mailbox, I looked at CPAN to see what modules were available for dealing with IMAP and mbox file formats. It didn't take long before I had a script that would upload a user's messages. That worked great for the mbox mailspool files- there was one file per user.

Next were the IMAP/webmail mailboxes. I had copied them all into separate folders for each user, so to automate it, I needed a script that could go into each user folder and upload the messages from each of the mailboxes it found there. However, there were some mailboxes I didn't want to upload (e.g. Spam, Virus, Trash boxes). Others, I needed to rename to go into Scalix equivalents.

The following Perl script is what I came up with.

#!/usr/bin/perl

use Smart::Comments;
use Mail::IMAPClient;
use Mail::Box::Mbox;

my $users = [
    {   username   =>  'user1',
        password   =>  'password1', },
    {   username   =>  'user2',
        password   =>  'password2', }, ];

my $folder_translation = {
    'sent-mail'     =>  'Sent Items',
    'sentmail'      =>  'Sent Items',
    'saved-drafts'  =>  'Drafts',
    'saved-messages'=>  'Drafts',
};

my $ignore_folders = [ 
    'spam-mail', 
    'virus-mail', 
    'mail-trash', 
    'Junk',
    'Junk E-mail', 
    'Trash', ];

foreach my $user (@$users) { 
    if( -d "/export/imap/$user->{'username'}") {

        warn "Looking at $user->{'username'}\n";
        my $imap = Mail::IMAPClient->new(  
            Server => 'mail.example.com',
            User    => $user->{'username'} . '@example.com',
            Password=> $user->{'password'},) or 
            die "Could not log in as $user->{'username'}";

       
        opendir DH, "/export/imap/$user->{'username'}";
        my @files = readdir DH;

        FILELOOP:
        foreach my $file (@files) {
            if($file !~ m/^\./) {   # Skip hidden
                foreach my $ign (@$ignore_folders) {
                    if($file eq $ign) {
                        warn "Ignoring $file\n";
                        next FILELOOP;
                    }
                }
                my $translated_name = $file;
                foreach my $tr_key (keys %$folder_translation) {
                    if($tr_key eq $file) {
                        $translated_name = $folder_translation->{$tr_key};
                    }
                }

                # Ready to upload messages
                warn "Uploading $file to $translated_name\n";

                my $folder = Mail::Box::Mbox->new(
                    folder => '/export/imap/' . $user->{'username'} .  '/' . $file);

                my @folders = $imap->folders();
                if(! grep /^$translated_name$/, @folders) {
                    $imap->create($translated_name);
                }

                foreach my $msg ($folder->messages) { ### Uploading msgs |===[%]           |
                    my $uid = $imap->append($translated_name, $msg->string);
                    if(! $uid) {
                        warn "Could not append message\n";
                    }
                }

                $folder->close() or warn "Could not close Mbox connection $@\n";
            }
        }
        closedir DH;

        $imap->disconnect() or warn "Could not close IMAP connection $@\n";
    }
    else {
        next;
    }

}

Last night, a close, personal friend sent me e-mail asking me for a "script fu" favor. It would seem that my close, personal friend had somehow acquired a collection of audio files and these files were in a format that his personal media player device would not play. The audio files were encoded in the MPEG-4 Audio (M4A) format and my close, personal friend's personal media player device supports a wide range of formats including FLAC, WAV, Ogg Vorbis, MP3, and perhaps some others I can't remember at the moment. My friend (who is my close, personal friend) asked me if I could "whip something up" that could convert all his files to MP3 format.

What a nice challenge!

For a few moments, I considered tackling this problem with a shell script using time-tested command line utilities like find, sed, and grep, but ultimately, I decided to engage this challenge using Perl.

I chose Perl over shell scripting mostly because the directory tree that needed to be traversed to access all the files had file and directory entries that contained an arbitrary number of whitespace characters and other not-so-friendly-to-shell characters. While I'm sure this could have been accomodated, it didn't seem like fun and I got excited thinking about how this could be handled with Perl.

It's a fun exercise to write Perl scripts that use opendir, readdir, and other standard Perl functions to interact with the host filesystem, but I knew there were some valuable CPAN modules, maybe even some "indistinguishable from magic" modules maintained by Damian Conway, I could use.

The first module I decided to use was File::Find::Rule which provides an alternative interface to File::Find.

use File::Find::Rule;

...

my @files = File::Find::Rule->file()
                            ->name('*.m4a')
                            ->in( '/path/to/root/of/files' );

File::Find is somewhat of a relic in terms of how it operates. It doesn't provide any kind of object oriented interface for using it and requires the user to pass subroutine references which isn't very pretty. File::Find::Rule, on the other hand, works relatively nicely.

To convert the files, once we had them in a list, I figured we'd use the veritable bastion of audio versatility that is mplayer and use its built-in ability to construct standalone WAV files from media files. To do this at the command line, use the pcm audio output option and specify a filename:

mplayer -ao pcm:file=myfile.wav someotherfile.m4a

There are a couple quadrillion other options and parameters you could also add, but this is the general gist of it.

After each MPEG-4 audio file is decoded and dumped into a WAV file, we can use lame to encode the WAV to MP3 format.

The lame utility, in its simplest form, works like this:

lame myfile.wav output.mp3

Like mplayer, there are a ridiculous number of options, switches, parameters, chants, and secret handshakes you can provide to make lame do its job faster, slower, on one foot, etc.

One of the, uhm... inconveniences, yeah, of calling other programs from a Perl script is that it isn't easy to tell what's going on or how things went on... or off, or whatever. The same is generally true in a shell scripting environment, but that's not important right now. What is important is that our good man Damian has done a fantastic job of helping make this easier by providing the Perl6::Builtins module to the Perl community. Apparently, this incongruent behavior when calling external applications is not an issue in the long-forthcoming next major version of Perl (Perl 6). Damian has just ported the nice behavior back to Perl 5.

The Perl6::Builtins module gives us a new system function we can use which behaves like a good system function should.

use Perl6::Builtins qw(system);

...

system('/usr/bin/mplayer', '-ao', 'pcm:file=/tmp/out.wav', $file) or 
   die "Could not dump $file to WAV: $!";

Using the standard system function, the above code would almost always result in a call to die because the normal exit status of the system call, while sensibly being zero because there are no errors during program execution, means something else entirely to Perl. Instead, Perl detects failure.

With Damian's indistinguishable-from-magic help, sanity is restored.

So, below is the whole script, with some minor things changed to protect the... uhm... lonely.

I'm not proud of the code I wrote to create the destination paths. It works, but not gracefully.

#!/usr/bin/perl

use Readonly;
use File::Find::Rule;
use Perl6::Builtins qw/system/;

Readonly my $sourcetree = 
    '/home/friend/audio/Zarry Lotter (Cantonese)';
Readonly my $sourcetree_exp = 
    '\/home\/friend/audio\/Zarry Lotter \(Cantonese\)';
Readonly my $desttree => 
    '/home/friend/audio/zarry_lotter_cantonese_mp3';

my @files = File::Find::Rule->file()
                            ->name('*.m4a')
                            ->in( $sourcetree );

if( ! -d $desttree) {
    mkdir $desttree || die "Could not make directory: $!";
}

foreach my $file (@files) {
    my $dest = $file;
    $dest =~ s{$sourcetree_exp}{$desttree};
    $dest =~ s{m4a}{mp3};

    my @path_components = split /\//, $dest;
    # Remove common leading components
    for (1 .. 5) { 
        shift @path_components ; 
    }
    # Remove filename
    pop @path_components;

    my $path = $desttree;
    foreach my $comp (@path_components) {
        if(!  -d "$path/$comp") {
            warn "Making directory [$path/$comp]";
            mkdir "$path/$comp";
        }
        $path = "$path/$comp";
    }

    # Use mplayer to dump file to WAV
    system('/usr/bin/mplayer', '-ao', 'pcm:file=/tmp/out.wav', $file) or 
        die "Could not dump $file to WAV: $!";

    # Use lame to make an mp3
    system('/usr/bin/lame', '/tmp/out.wav', $dest) or 
        die "Could not convert $file to MP3: $!";
}

It doesn't matter what language you do your server-side web development in, presentation templates just make so much sense. Here are some reasons why:

• Templates (usually) let you re-use commonly used blocks of code.
• Templates (usually) make it easy (or easier) for a web designer (e.g. not a developer) to work on the presentation layout of your application.
• Templates allow you to separate presentation from business logic and will help you separate them in how you think of your application as well.

By templates, I mean files that contain HTML data along with some special coded method of interpolating dynamic data into the HTML.

By this definition, PHP and ASP code are template languages themselves. That's one of the more substantial reasons I've come to dislike these languages! You'll come to understand as you read more below.

Here's a very simple template example:

<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>$TITLE</title>
    </head>
    <body>
        <h1>Welcome to the $SITE_NAME website!</h1>
    </body>
</html>

The above example uses the syntax defined by the CGI::FastTemplate Perl module. I used this module for a couple of projects in 2000 or so and while it does make it easy to design a page's look and feel around the dynamic content your web application is going to drop into the page, it works best when your content doesn't need to be nested inside HTML. For example, dynamics data in a table does not work well with CGI::FastTemplate. Neither do lists of data.

Some templating systems, like CGI::FastTemplate to some extent, can handicap you too much. Others, like HTML::Embperl or ePerl, in my opinion, provide too much capability at the template level.

You don't want to be able to write your entire application in a template. You might as well be writing your application in PHP, JSP, or ASP (or Apache::ASP).

I think a templating system should provide you with just enough logic so that you can affect how data is presented and not much more logic than that. I'm not alone in thinking these things. See Wikipedia topics: Web template system, Model-view-controller, and Separation of concerns.

So, as a result of study of templating systems for Perl, I recommend Template (Template Toolkit for Perl).

Note: Template Toolkit does allow some things I don't think have any place in templates, but it doesn't do so by default. You have to make a conscious choice to do those things, read the documentation, etc. and that should (hopefully) discourage you from doing so.

How to use Template Toolkit

Let's take a look at how Template Toolkit would be used to mimic the CGI::FastTemplate example above.

<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>[% title %]</title>
    </head>
    <body>
        <h1>Welcome to the [% site_name %] website!</h1>
    </body>
</html>

It's not a whole lot different, is it? And, actually, it's a little more typing to key in those beginning and ending brackets instead of just a preceding dollar sign.

Now, here's an example of how this template file (we'll call it page1.tt) would be used in Perl code, say, in a CGI script.

#!/usr/bin/perl

use CGI qw/:standard/;
use Template;

my $q = CGI;
my $tt = Template->new({
    INCLUDE_PATH    =>  '/var/www/templates/'});

my $vars = {};

$vars->{'title'} = 'Joe Schmoe Shoe Repair - Home';
$vars->{'site_name'} = 'Joe Schmoe Shoe Repair';

print $q->header('text/html');
print $tt->process('page1.tt', $vars) || 
    die $tt->error();

Great! Now we know Template is just as good as CGI::FastTemplate! What else can it do?!

Wrapping an application

One thing I often do with Template Toolkit is use it to wrap applications so that every page has the same look and feel. You do this by defining a wrapper template in the hash reference you pass to the Template constructor:

my $tt = Template->new({
    INCLUDE_PATH    =>  '/var/www/templates/',
    WRAPPER         =>  'sitewrapper.tt', });

Now every page generated by the process() function will include the contents of sitewrapper.tt around it.

The wrapper template needs to contain a special template directive in it: [% content %]. This is where the contents of your specified templates are placed.

Here is an example of a wrapper template:

<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>[% template.title %]</title>
        <link rel="stylesheet" type="text/css" href="style.css" />
    </head>
    <body>
        <div class="sidebar">[% INCLUDE sidebar.tt %]</div>
        <div class="body">[% content %]</div>
        <div class="footer">[% INCLUDE footer.tt %]</div>
    </body>
</html>

Notice the use of the [% content %] directive in the wrapper template.

There are a couple other interesting uses of Template Toolkit directives in this example, notably the INCLUDE directive (for inserting the parsed output of other templates) and a special directive within the HTML title tags.

The [% template.title %] directive should make sense once we see how a normal template might look when using a wrapper:

    [% META title = "Contact us" %]
    <h1>Contacting Joe Schmoe Shoe Repair!</h1>
    <p>See the list below for our telephone numbers:</p>
    <ul>
    [% FOREACH phone = phone_numbers %]
        <li>[% phone.name %]: [% phone.number %]</li>
    [% END %]
    

The META directive let's us pass data into the wrapper template.

Flow control

The above template example also introduces to one of the reasons I love Template Toolkit: flow control. In this case, a for-each loop. The variable phone_numbers could be a list of hashes we get from a database and pass to the template. Each hash contains a name-value pair named name and one named number. The FOREACH directive allows us to iterate through the phone_numbers list and assign a reference the current hash to a variable we've named phone. Inside the loop, we can reference the individual name-value pairs by name.

This is only one of the flow control structures Template Toolkit offers. There are also IF-THEN-ELSIF-ELSE structures, CASE/SWITCH structures, and WHILE loops.

These should give you just a taste of what Template Toolkit can do. With the Template Perl module installed on your system, you get all the documentation you could hope for as POD and man files. The Template Toolkit website also has all this documentation online as well.

Below is a bit of a more complex template I created years ago for the utahisps.com website.

      <h3>[% company.name %]</h3>
      <table cellpadding="6">
        <tr>
          <td>Services</td>
          <td>
                [% has_services = 0 %]
                [% FOREACH service_name = services.keys %]
                  [% IF services.$service_name %]
                    <a href="/isp/[% company.company_id %]/[% service_name %]">
                    [ % service_name %]</a><br/>
                    [% has_services = 1 %]
                  [% END %]
                [% END %]
                [% IF has_services == 0 %]
                  N/A
                [% END %]
                  <!-- None listed -->

              </td>
            </tr>
        <tr>
          <td>Address</td>
          <td>
            [% company.addr1 %]<br/>
        [% IF company.addr2.length %]
        [% company.addr2 %]<br/>
        [% END %]
        [% company.city %], [% company.state %] [% company.zip %]
          </td>
        </tr>
        <tr>
          <td>Phones</td>
          <td>
            [% IF company.phone1.length %]
                [% company.phone1 %] ([% company.phone1_desc %])
        [% END %]
            [% IF company.phone2.length %]
        <br/>
                [% company.phone2 %] ([% company.phone2_desc %])
        [% END %]
          </td>
        </tr>
        [% IF company.fax.length %]
        <tr>
          <td>Fax</td>
          <td>[% company.fax %]</td>
            </tr>
            [% END %]
            <tr>
              <td>Website</td>
          <td>< <a href="[% company.www_url %]" 
          target="_new">[% company.www_url %]
          </a> ></td>
            </tr>
        [% IF company.info_email.length %]
        <tr>
          <td>Info E-mail</td>
              <td>< <a href="mailto:[% company.info_email %]"
              target="_new">
              [% company.info_email %]</a> ></td>
            </tr>
            [% END %]
        <tr>
          <td>Payment</td>
              <td>
                [% IF company.credit_cards.length %]
                  [% company.credit_cards %]
                [% ELSE %]
                  Cash/check only
                [% END %]
            </tr>
        [% IF company.oper_since %]
        <tr>
          <td>Oper since</td>
              <td>[% company.oper_since %]</td>
            </tr>
            [% END %]

        <tr>
              <td><a href="http://www.angio.net/rep/" 
              target="_new">Utah
              REP</a><br/>member</td>  
              <td>[% IF company.ut_rep %]Yes[% ELSE %]No[% END %]</td>
            </tr>
            
        [% IF company.os.length %]
        <tr>
          <td>Primary OS</td>
              <td>[% company.os %]</td>
            </tr>
            [% END %]

        [% IF company.upstreams %]
        <tr>
          <td>Upstream feeds</td>
              <td>[% company.upstreams %]</td>
            </tr>
            [% END %]


        [% IF company.notes.length %]
        <tr>
          <td>Notes</td> 
              <td><tt>[% company.notes %]</tt></td>
            </tr>
            [% END %]
     </table>