Listen to any kind of syndicated talk radio program and you'll usually hear about some companion website the program has. Usually, there are a handful of free things you can get on a program's website, but many of these sites have a pay-to-play members' area where the really good content is. This includes MP3 downloads of the shows, access to live audio and/or video streams, special behind-the-scenes content, forums, desktop backgrounds, etc.

The MP3 downloads are very convenient for people who don't have the luxury of sitting in front of a radio (or driving a car) for a solid three hours while a radio program is broadcast (with advertisements). It's also a boon for people who find radio advertisements annoying.

The only problem with the MP3 downloads is that theme music and produced portions of the program can not, by law, be included in the MP3 file because otherwise the MP3 would be a copyright violation.

Live streams, on the other hand, are not subject to the above described restriction because they're like a broadcast in nature. They're not a time-shift of the original program. So, if you listen to the live stream or even listen to a pre-recorded program as a stream, music and produced segments may be included.

I listen to the Glenn Beck radio program quite often. I used to download the MP3 files to listen to in the car, but it got annoying everytime Glenn and his producers would put together a segment like "Sportscasters at the 2031 animal-human hybrid baseball games", or "The History Of the Democratic Superdelegates" and I would hear Glenn say, "Listen to this... [pause] Oh man! That was great! Wasn't that great, Stu? Oh yeah! Alright! Dan? Wasn't that just the best? Yeah. Oh yeah."

I decided I needed to figure out how to save a stream.

I knew it was possible. Lots of software applications exist for any operating systems that will convert audio from a live stream into a static WAV file or similar. The open source program mplayer is one such example.

Breaking it down

First of all, I needed to figure out how the stream content made its way to my computer.

After I've logged into the Glenn Beck website as an Insider, I can click a link to listen to a stream of a particular hour of the program (or the whole program) in Windows Media format or RealAudio format. I figured I'd have better luck extracting the audio from the Windows Media format, so I went that route. Instead of just clicking the link and letting my web browser find some program that could handle the content, I saved the content to a file and then looked at the file.

The file it saved was a fairly straightforward XML file that looked something like this:

<ASX VERSION="3.0">
  <TITLE>Glenn Beck</TITLE>
  <AUTHOR>Premiere Radio Networks</AUTHOR>
  <COPYRIGHT>Copyright 2008</COPYRIGHT>

 <ENTRY>

    <TITLE>Glenn Beck 1</TITLE>

    <AUTHOR>Premiere Radio Networks</AUTHOR>

    <COPYRIGHT>Copyright 2008</COPYRIGHT>
 

    <REF HREF="mms://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603.WMA?auth=blahblahblahblahblah" />

    <REF HREF="http://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603.WMA?auth=blahblahblahblahblahblah
  </ENTRY>

  <ENTRY>

    <TITLE>Glenn Beck 2</TITLE>

    <AUTHOR>Premiere Radio Networks</AUTHOR>

    <COPYRIGHT>Copyright 2008</COPYRIGHT>

    

    <REF HREF="mms://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603_CLIP01.WMA?auth=blahblahblahblahblahblah" />

    <REF HREF="http://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603_CLIP01.WMA?auth=blahblahblahblahandblah" />

  </ENTRY>

...and so on.

This XML defines the MMS URLs for each segment of the show. There are several segments each hour. These individual MMS URLs are what I needed to feed to the application that was going to convert the audio stream to a file. In my case, I decided to use mplayer because it's just so good at everything it does!

The command line for doing the stream-to-file conversion looks like this:

mplayer -vc null -vo null -ao pcm:fast:file=dumpfile.wav \
    'mms://a0011.v67134.c6713.g.vm.akamaistream.net/blahblahblah...'

The real magic in the above command is where I use -ao pcm to tell mplayer to use the PCM file writer audio output driver (instead of sending the audio to my speakers).

This gives me a WAV file which I'll want to convert to an MP3 or Ogg-Vorbis file.

To convert a WAV file generated by the mplayer command above to an MP3 file, I use the open source lame tool:

lame -mf -q2 dumpfile.wav GlennBeck.mp3

Or, convert it to Ogg-Vorbis (the completely open and better-sounding-than-MP3 lossy audio codec):

oggenc -q2 --downmix -o GlennBeck.ogg dumpfile.wav

I've now covered the basic mechanical components of converting an audio stream into an MP3 or Ogg-Vorbis file. Next I automate it all.

Automation

Because I'm a long-time Perl junkie, I investigated how I could use a Perl script to act as the glue between the components and get the whole process of capturing a stream and converting it to MP3 or Ogg-Vorbis.

In the above walk-through, I manually logged into the Glenn Beck website with my web browser. To really completely automate this puppy, I wanted the script to log in for me. It didn't take me very long to figure out the Perl CPAN module WWW::Mechanize was what I needed to use.

WWW::Mechanize does several handy things for the programmer. It loads and parses web pages and can follow links, populate forms, and other basic kinds of interaction. It keeps track of its own cookies and session data too.

To get into the Insider area of the Glenn Beck website, members must enter their username and password on the Insider login page.

Looking at the HTML source for this page, I learned the form was named "aform", the username field was named "iUName", and the password field was named "iPassword".

I now had all the information I needed for WWW::Mechanize to log in:

my $agent = WWW::Mechanize->new(
    cookie_jar  => {},
);
   
my $resp = $agent->get('http://www.glennbeck.com/content/insider');
   
if($resp->is_success) {
    $resp = $agent->submit_form(
        form_name   =>  'aform',
        fields      =>  {   'iUName'    =>  'myusername',
                                'iPassword' =>  'shhhhhhhh!', },
        button      =>  'submit');

Walking through the code above: First, I create the WWW::Mechanize object with an in-memory cookie jar (cookie_jar => {}). Next, I use the object to get() the log-in page. If everything works well so far, I tell the object to find the form named "aform", fill in the username and password fields, and submit the form.

One thing I realized as I was debugging my script was that after I logged in on the Insider page, I was immediately redirected to another page. In order for my script to work, it needed to follow the redirect. This was an easy fix:

my $agent = WWW::Mechanize->new(
    cookie_jar  => {},
    redirect_ok => 1,
);

The page I got redirected to has the links on it for the streaming audio, so I'm exactly where I want to be if I want to capture and convert the latest and greatest Glenn Beck Program audio stream.

WWW::Mechanize can find links within the page with a variety of methods. One of these leverages Perl's excellent support for regular expressions. You can also search for links by the order in which they appear. The link I'm looking for looks like this:

<a href="http://www.premiereinteractive.com/cgi-bin/members.cgi?stream=shows/GLENNBECKWIN20080604&site=glennbeck&type=win_show"><img src="http://media.glennbeck.com/images/common/header_media5off.jpg" name="icon5" width="26" height="34" border="0" id="icon5" onMouseOver="MM_swapImage('icon5','','http://media.glennbeck.com/images/common/header_media5on.jpg',1)" onMouseOut="MM_swapImgRestore()" /></a>

So, my script has the following:

$link = $agent->find_link( url_regex => qr/${datestr}.*win_show$/);
$resp = $agent->get($link);

This assumes I have a scalar variable $datestr that contains a formatted date for the show I want to capture.

Originally, I was going to use one of Perl's several XML-parsing modules to make sense of the XML in the stream link, but in the end all I needed was a regular expression to extract the mms: URLs.

my $xml = $resp->decoded_content;
my (@urls) = $xml =~ m/HREF="(mms:[^"]+)"/msg;

This gives me a list of URLs stored in @urls. Now I just need to feed them to mplayer:

$i = 1;
foreach my $u (@urls) {
    my $seq = sprintf("%02d", $i);
    my @cmd = ( 'mplayer', 
            '-vc', 'null', 
            '-vo', 'null',
            '-ao', "pcm:fast:file=${datestr}-${seq}.wav", 
            $u);
    system(@cmd);
    if ($? == -1) {
        print "failed to execute: $!\n";
    }
    elsif ($? & 127) {
        printf "child died with signal %d, %s coredump\n",
        ($? & 127),  ($? & 128) ? 'with' : 'without';
    }
    else {
        printf "child exited with value %d\n", $? >> 8;
    }

    $i++;
}

This little ditty creates an output file for each of the segment streams. These are named something like 20080604-05.wav.

When the loop is finished, I have several WAV files sitting on the disk. Now I need to somehow sew them all together into one big WAV file so I can convert it to an MP3 or Ogg-Vorbis file. For this, I turn to sox. I decided to have the Perl script generate a shell script to run all the sox and lame commands needed.

open FH, ">/tmp/${datestr}.sh";
foreach my $j (1..($i-1)) {
    my $seq = sprintf("%02d", $j);
    print FH 'sox ', "${datestr}-${seq}.wav", " -t raw - | cat >> /tmp/${datestr}.raw", "\n";
}
print FH 'sox -w -s -c 1 -r 22050 ', "/tmp/${datestr}.raw ${datestr}.wav\n";
print FH "lame -mf -q2 ${datestr}.wav ${datestr}.mp3 ";
print FH "--tt \"Glenn Beck Show - $datestr\" ";
print FH "--ta \"Glenn Beck\" --add-id3v2\n";
close FH;

Then, I run the shell script:

system('sh', "/tmp/${datestr}.sh");

Finally, I do a little cleanup:

unlink "/tmp/${datestr}.sh", "/tmp/${datestr}.raw", map({"${datestr}-$_.wav"} (1..($i-1)));

And, I'm done. There are many other ways I could have gone about doing this, but I found a way that worked and ran with it. I'd love to hear from people who have done something similar and how they did it.