July 2008 Archives

Last night, a close, personal friend sent me e-mail asking me for a "script fu" favor. It would seem that my close, personal friend had somehow acquired a collection of audio files and these files were in a format that his personal media player device would not play. The audio files were encoded in the MPEG-4 Audio (M4A) format and my close, personal friend's personal media player device supports a wide range of formats including FLAC, WAV, Ogg Vorbis, MP3, and perhaps some others I can't remember at the moment. My friend (who is my close, personal friend) asked me if I could "whip something up" that could convert all his files to MP3 format.

What a nice challenge!

For a few moments, I considered tackling this problem with a shell script using time-tested command line utilities like find, sed, and grep, but ultimately, I decided to engage this challenge using Perl.

I chose Perl over shell scripting mostly because the directory tree that needed to be traversed to access all the files had file and directory entries that contained an arbitrary number of whitespace characters and other not-so-friendly-to-shell characters. While I'm sure this could have been accomodated, it didn't seem like fun and I got excited thinking about how this could be handled with Perl.

It's a fun exercise to write Perl scripts that use opendir, readdir, and other standard Perl functions to interact with the host filesystem, but I knew there were some valuable CPAN modules, maybe even some "indistinguishable from magic" modules maintained by Damian Conway, I could use.

The first module I decided to use was File::Find::Rule which provides an alternative interface to File::Find.


use File::Find::Rule;

...

my @files = File::Find::Rule->file()
                            ->name('*.m4a')
                            ->in( '/path/to/root/of/files' );

File::Find is somewhat of a relic in terms of how it operates. It doesn't provide any kind of object oriented interface for using it and requires the user to pass subroutine references which isn't very pretty. File::Find::Rule, on the other hand, works relatively nicely.

To convert the files, once we had them in a list, I figured we'd use the veritable bastion of audio versatility that is mplayer and use its built-in ability to construct standalone WAV files from media files. To do this at the command line, use the pcm audio output option and specify a filename:


mplayer -ao pcm:file=myfile.wav someotherfile.m4a

There are a couple quadrillion other options and parameters you could also add, but this is the general gist of it.

After each MPEG-4 audio file is decoded and dumped into a WAV file, we can use lame to encode the WAV to MP3 format.

The lame utility, in its simplest form, works like this:


lame myfile.wav output.mp3

Like mplayer, there are a ridiculous number of options, switches, parameters, chants, and secret handshakes you can provide to make lame do its job faster, slower, on one foot, etc.

One of the, uhm... inconveniences, yeah, of calling other programs from a Perl script is that it isn't easy to tell what's going on or how things went on... or off, or whatever. The same is generally true in a shell scripting environment, but that's not important right now. What is important is that our good man Damian has done a fantastic job of helping make this easier by providing the Perl6::Builtins module to the Perl community. Apparently, this incongruent behavior when calling external applications is not an issue in the long-forthcoming next major version of Perl (Perl 6). Damian has just ported the nice behavior back to Perl 5.

The Perl6::Builtins module gives us a new system function we can use which behaves like a good system function should.


use Perl6::Builtins qw(system);

...

system('/usr/bin/mplayer', '-ao', 'pcm:file=/tmp/out.wav', $file) or 
   die "Could not dump $file to WAV: $!";

Using the standard system function, the above code would almost always result in a call to die because the normal exit status of the system call, while sensibly being zero because there are no errors during program execution, means something else entirely to Perl. Instead, Perl detects failure.

With Damian's indistinguishable-from-magic help, sanity is restored.

So, below is the whole script, with some minor things changed to protect the... uhm... lonely.

I'm not proud of the code I wrote to create the destination paths. It works, but not gracefully.


#!/usr/bin/perl

use Readonly;
use File::Find::Rule;
use Perl6::Builtins qw/system/;

Readonly my $sourcetree = 
    '/home/friend/audio/Zarry Lotter (Cantonese)';
Readonly my $sourcetree_exp = 
    '\/home\/friend/audio\/Zarry Lotter \(Cantonese\)';
Readonly my $desttree => 
    '/home/friend/audio/zarry_lotter_cantonese_mp3';

my @files = File::Find::Rule->file()
                            ->name('*.m4a')
                            ->in( $sourcetree );

if( ! -d $desttree) {
    mkdir $desttree || die "Could not make directory: $!";
}

foreach my $file (@files) {
    my $dest = $file;
    $dest =~ s{$sourcetree_exp}{$desttree};
    $dest =~ s{m4a}{mp3};

    my @path_components = split /\//, $dest;
    # Remove common leading components
    for (1 .. 5) { 
        shift @path_components ; 
    }
    # Remove filename
    pop @path_components;

    my $path = $desttree;
    foreach my $comp (@path_components) {
        if(!  -d "$path/$comp") {
            warn "Making directory [$path/$comp]";
            mkdir "$path/$comp";
        }
        $path = "$path/$comp";
    }

    # Use mplayer to dump file to WAV
    system('/usr/bin/mplayer', '-ao', 'pcm:file=/tmp/out.wav', $file) or 
        die "Could not dump $file to WAV: $!";

    # Use lame to make an mp3
    system('/usr/bin/lame', '/tmp/out.wav', $dest) or 
        die "Could not convert $file to MP3: $!";
}

It doesn't matter what language you do your server-side web development in, presentation templates just make so much sense. Here are some reasons why:

  • Templates (usually) let you re-use commonly used blocks of code.
  • Templates (usually) make it easy (or easier) for a web designer (e.g. not a developer) to work on the presentation layout of your application.
  • Templates allow you to separate presentation from business logic and will help you separate them in how you think of your application as well.

By templates, I mean files that contain HTML data along with some special coded method of interpolating dynamic data into the HTML.

By this definition, PHP and ASP code are template languages themselves. That's one of the more substantial reasons I've come to dislike these languages! You'll come to understand as you read more below.

Here's a very simple template example:


<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>$TITLE</title>
    </head>
    <body>
        <h1>Welcome to the $SITE_NAME website!</h1>
    </body>
</html>

The above example uses the syntax defined by the CGI::FastTemplate Perl module. I used this module for a couple of projects in 2000 or so and while it does make it easy to design a page's look and feel around the dynamic content your web application is going to drop into the page, it works best when your content doesn't need to be nested inside HTML. For example, dynamics data in a table does not work well with CGI::FastTemplate. Neither do lists of data.

Some templating systems, like CGI::FastTemplate to some extent, can handicap you too much. Others, like HTML::Embperl or ePerl, in my opinion, provide too much capability at the template level.

You don't want to be able to write your entire application in a template. You might as well be writing your application in PHP, JSP, or ASP (or Apache::ASP).

I think a templating system should provide you with just enough logic so that you can affect how data is presented and not much more logic than that. I'm not alone in thinking these things. See Wikipedia topics: Web template system, Model-view-controller, and Separation of concerns.

So, as a result of study of templating systems for Perl, I recommend Template (Template Toolkit for Perl).

Note: Template Toolkit does allow some things I don't think have any place in templates, but it doesn't do so by default. You have to make a conscious choice to do those things, read the documentation, etc. and that should (hopefully) discourage you from doing so.

How to use Template Toolkit

Let's take a look at how Template Toolkit would be used to mimic the CGI::FastTemplate example above.


<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>[% title %]</title>
    </head>
    <body>
        <h1>Welcome to the [% site_name %] website!</h1>
    </body>
</html>

It's not a whole lot different, is it? And, actually, it's a little more typing to key in those beginning and ending brackets instead of just a preceding dollar sign.

Now, here's an example of how this template file (we'll call it page1.tt) would be used in Perl code, say, in a CGI script.


#!/usr/bin/perl

use CGI qw/:standard/;
use Template;

my $q = CGI;
my $tt = Template->new({
    INCLUDE_PATH    =>  '/var/www/templates/'});

my $vars = {};

$vars->{'title'} = 'Joe Schmoe Shoe Repair - Home';
$vars->{'site_name'} = 'Joe Schmoe Shoe Repair';

print $q->header('text/html');
print $tt->process('page1.tt', $vars) || 
    die $tt->error();

Great! Now we know Template is just as good as CGI::FastTemplate! What else can it do?!

Wrapping an application

One thing I often do with Template Toolkit is use it to wrap applications so that every page has the same look and feel. You do this by defining a wrapper template in the hash reference you pass to the Template constructor:


my $tt = Template->new({
    INCLUDE_PATH    =>  '/var/www/templates/',
    WRAPPER         =>  'sitewrapper.tt', });

Now every page generated by the process() function will include the contents of sitewrapper.tt around it.

The wrapper template needs to contain a special template directive in it: [% content %]. This is where the contents of your specified templates are placed.

Here is an example of a wrapper template:


<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>[% template.title %]</title>
        <link rel="stylesheet" type="text/css" href="style.css" />
    </head>
    <body>
        <div class="sidebar">[% INCLUDE sidebar.tt %]</div>
        <div class="body">[% content %]</div>
        <div class="footer">[% INCLUDE footer.tt %]</div>
    </body>
</html>

Notice the use of the [% content %] directive in the wrapper template.

There are a couple other interesting uses of Template Toolkit directives in this example, notably the INCLUDE directive (for inserting the parsed output of other templates) and a special directive within the HTML title tags.

The [% template.title %] directive should make sense once we see how a normal template might look when using a wrapper:


    [% META title = "Contact us" %]
    <h1>Contacting Joe Schmoe Shoe Repair!</h1>
    <p>See the list below for our telephone numbers:</p>
    <ul>
    [% FOREACH phone = phone_numbers %]
        <li>[% phone.name %]: [% phone.number %]</li>
    [% END %]
    

The META directive let's us pass data into the wrapper template.

Flow control

The above template example also introduces to one of the reasons I love Template Toolkit: flow control. In this case, a for-each loop. The variable phone_numbers could be a list of hashes we get from a database and pass to the template. Each hash contains a name-value pair named name and one named number. The FOREACH directive allows us to iterate through the phone_numbers list and assign a reference the current hash to a variable we've named phone. Inside the loop, we can reference the individual name-value pairs by name.

This is only one of the flow control structures Template Toolkit offers. There are also IF-THEN-ELSIF-ELSE structures, CASE/SWITCH structures, and WHILE loops.

These should give you just a taste of what Template Toolkit can do. With the Template Perl module installed on your system, you get all the documentation you could hope for as POD and man files. The Template Toolkit website also has all this documentation online as well.

Below is a bit of a more complex template I created years ago for the utahisps.com website.


      <h3>[% company.name %]</h3>
      <table cellpadding="6">
        <tr>
          <td>Services</td>
          <td>
                [% has_services = 0 %]
                [% FOREACH service_name = services.keys %]
                  [% IF services.$service_name %]
                    <a href="/isp/[% company.company_id %]/[% service_name %]">
                    [ % service_name %]</a><br/>
                    [% has_services = 1 %]
                  [% END %]
                [% END %]
                [% IF has_services == 0 %]
                  N/A
                [% END %]
                  <!-- None listed -->

              </td>
            </tr>
        <tr>
          <td>Address</td>
          <td>
            [% company.addr1 %]<br/>
        [% IF company.addr2.length %]
        [% company.addr2 %]<br/>
        [% END %]
        [% company.city %], [% company.state %] [% company.zip %]
          </td>
        </tr>
        <tr>
          <td>Phones</td>
          <td>
            [% IF company.phone1.length %]
                [% company.phone1 %] ([% company.phone1_desc %])
        [% END %]
            [% IF company.phone2.length %]
        <br/>
                [% company.phone2 %] ([% company.phone2_desc %])
        [% END %]
          </td>
        </tr>
        [% IF company.fax.length %]
        <tr>
          <td>Fax</td>
          <td>[% company.fax %]</td>
            </tr>
            [% END %]
            <tr>
              <td>Website</td>
          <td>< <a href="[% company.www_url %]" 
          target="_new">[% company.www_url %]
          </a> ></td>
            </tr>
        [% IF company.info_email.length %]
        <tr>
          <td>Info E-mail</td>
              <td>< <a href="mailto:[% company.info_email %]"
              target="_new">
              [% company.info_email %]</a> ></td>
            </tr>
            [% END %]
        <tr>
          <td>Payment</td>
              <td>
                [% IF company.credit_cards.length %]
                  [% company.credit_cards %]
                [% ELSE %]
                  Cash/check only
                [% END %]
            </tr>
        [% IF company.oper_since %]
        <tr>
          <td>Oper since</td>
              <td>[% company.oper_since %]</td>
            </tr>
            [% END %]

        <tr>
              <td><a href="http://www.angio.net/rep/" 
              target="_new">Utah
              REP</a><br/>member</td>  
              <td>[% IF company.ut_rep %]Yes[% ELSE %]No[% END %]</td>
            </tr>
            
        [% IF company.os.length %]
        <tr>
          <td>Primary OS</td>
              <td>[% company.os %]</td>
            </tr>
            [% END %]

        [% IF company.upstreams %]
        <tr>
          <td>Upstream feeds</td>
              <td>[% company.upstreams %]</td>
            </tr>
            [% END %]


        [% IF company.notes.length %]
        <tr>
          <td>Notes</td> 
              <td><tt>[% company.notes %]</tt></td>
            </tr>
            [% END %]
     </table>

We're about one month away from the 2008 Utah Open Source Conference and it's shaping up to be a sweet conference. This year's conference will be Thursday, August 28 through Saturday, August 30 and will be held at Salt Lake Community College Redwood Campus.

Keynote presentations will be given by Paul Frields, Fedora Project Leader and Joe Brockmeier, OpenSUSE Community Leader.

Over 50 presentations by members open source community, both local and abroad, have been approved, including one by yours truly: Tools for video and images.

I should mention that I've been very involved with the planning and execution of this years conference and I've been named a "trustee" of the Utah Open Source Foundation. I am thrilled to be a part of this great organization!

This year adds many new things to UTOSC including "Family Day" (Saturday, 30 Aug) which is the day to bring your spouse and/or children. There will be plenty of presentations on Family Day geared at those just starting out with open source software and tackling conventional tasks like digital photos, desktop publishing, and genealogy. Plus, we're planning on having games and activities for the whole family.

If you have not already registered for the Utah Open Source Conference, it's not too late. If you register before August 7, you can qualify for the early bird rate of $70/person. If you are a member of a participating local user group, you can qualify for a discounted rate. Talk to your group leaders for more information.

Perl is over twenty years old, but it continues to evolve as a programming language. When Perl 6 is released to the world, we're going to see an incredibly modern language that is adaptable to tackle many of the challenges faced by programmers today including internationalization/localization, Internet networking, object orientation, and more.

But this blog post isn't about Perl 6. This is the first in a series of posts designed to help people ease into Perl as a web development language and is influenced largely by tutorials and seminars I've given in person and online about using Perl as a web development tool.

Perl was embraced for web development in the early and mid 1990s because it had already become a powerful language for text-processing and system administration. As such, it was a natural first choice for developing web applications.

Perl programmers at the time adapted public domain algorithms into Perl code to process HTML forms, but the results were not pretty.


my $hashRef = {};
my $buffer = "";

if ($ENV{'REQUEST_METHOD'} eq 'GET') {
    $buffer = $ENV{'QUERY_STRING'};
}
else {
    read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
}

foreach (split(/&/, $buffer)) {
    my($key, $value) = split(/=/, $_);
    $key   = decodeURL($key);
    $value = decodeURL($value);
    %{$hashRef}->{$key} = $value;
}

sub decodeURL {
    $_ = shift;
    tr/+/ /;
    s/%(..)/pack('c', hex($1))/eg;
    return($_);
}

No wonder Perl is often labeled as "unreadable!"

Thankfully, Lincoln Stein came along and gave us CGI.pm-- a Perl 5 module that makes several tasks the Perl web developer is commonly faced with, easier:

  • Generating good HTTP headers
  • Parsing form data
  • Generating HTML
  • Handling file uploads
  • Debugging HTML form processes
  • ...and more

Web developers often fail to realize they take on a greater level of responsibility when they leave the realm of static HTML pages, images, MP3 files, etc. served up by a HTTP server like Apache and enter the world of dynamic content generation, interactive web applications, and forms and form processors.

Web developers must write code to generate headers for the client that would otherwise be handled by the web server. This includes, at a minimum, a Content-type header, but may also include Content-length, Content-disposition, or Location. Developers must also provide the client with an HTTP status code (e.g. 200, 404, 302, etc.).

You may use CGI.pm using object-oriented syntax or not. I prefer to use the object-oriented syntax because it means I'm in step with other Perl modules I employ in my code.


use CGI;

my $q = new CGI;
print   $q->header('text/html'),
        $start_html(    -title =>   'React' );

my $answer = $q->param('foo');
if($answer eq 'bar') {
    print $q->p("You answered correctly!");
}
else {
    print $q->p("That is incorrect.");
}

print $q->end_html;

This example generates different content based on the value of a form variable (foo) passed in. If this script were named react.cgi then it might be called with a URL like react.cgi?foo=bar.

Notice the code above uses the header() function to generate a valid HTTP header. There are many options you may pass to this function but if you pass one scalar string as in this example, it will use that data to generate a Content-type header.

The above above demonstrates some of CGI.pm's HTML-generation capabilities (the use of the p() function to generate paragraph tags around content and the use of the start_html() and end_html() functions to properly form the beginning and end of an HTML document and generate the necessary title tags in the header.)

The above example also shows how you may use CGI.pm to access form variables (the param() function).

For a couple years, I used CGI.pm quite heavily in my web development tasks. I grew very attached to its HTML-generation capabilities and the fact that if your HTML is generated from CGI.pm code, it will always be valid, start tags will always have matching end tags, and so forth. For example, here is a snippet of code I wrote circa 1999-2000:


print   $q->start_form(
           -method=> 'POST',
            -action => 'add_event.cgi'),
         $q->hidden({-name=>'r'}),
         $q->table(
            $q->Tr(
                $q->td(
                    _event_form($q)),
                $q->td({
                    -valign=>'top',
                    -align=>'left'}),
                    $q->submit({
                        -name=>'c',
                        -value=>'Save Event Info'}),
                    $q->br,
                        $q->submit({
                            -name=>'c',
                            -value=>'Cancel'}))),
         $q->end_form;

There are definite advantages to writing code this way, but the disadvantages are numerous, especially if you intend for your application to be maintained by someone other than yourself. If you intend to have a web designer who has no Perl competence work on the layout, this type of code will have them running away, screaming, and flailing their arms wildly.

That being said, if you need to throw together a simple, self-encapsulated interactive web page, doing everything with CGI.pm isn't a bad way to go.

In most cases, CGI.pm is already installed on your Linux system. To read all about it, type perldoc CGI at your shell command prompt, or go to http://search.cpan.org/perldoc?CGI with your favorite web browser.

What's next?

Well, what's next partly depends on the type of feedback I get from this post, if any. Generally, I plan to talk about templating, database interaction, and more. But, if people have some specific things they would like to see my spin on, I'd be more than happy to oblige.

About this Archive

This page is an archive of entries from July 2008 listed from newest to oldest.

June 2008 is the previous archive.

August 2008 is the next archive.

Find recent content on the main index or look in the archives to find all content.