Recently in Open source software Category

There have been a couple things about Fedora 12 that haven’t been as nice as I would have liked. I finally solved one of them tonight.

My laptop, a Dell Latitude D830(N), has an NVidia Quadro NVS 140M video chipset in it. Fedora 12 worked out of the box with the open source nouveau driver which is an experimental reverse-engineered driver for NVidia chipsets. It works pretty well and I probably would have kept using it if I could get my laptop to hibernate properly. Instead, I could never get the laptop to come back to life after it went into hibernation.

Meanwhile, the RPMFusion folks (a popular third-party repository) usually have the kmod-nvidia package available to install which gives you everything you need to run the proprietary NVidia drivers (Fedora doesn’t include this because they adhere to an all-open, non-patent-encumbered package policy). However, the kmod-nvidia package wasn’t available for Fedora 12.

When it did show up on rpmfusion, there were some caveats. Fedora had done some work to make the nouveau driver work as seamlessly as possible and, as a result, made it a little more difficult to install the proprietary driver. The RPMFusion folks have some errata info on how to get the proprietary driver working. I’ll summarize the process here since it’s a little tricky to execute and understand.

Before you do anything the RPMFusion information says to do, you should obviously install the kmod-nvidia package. Then, run:

nvidia-system-config enable

(I usually do this with sudo.)

Then, reboot into runlevel 3 and proceed with the commands RPMFusion’s page recommends.

The first commands the RPMFusion info indicates should be run are these:

mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r)-nouveau.img
dracut /boot/initramfs-$(uname -r).img $(uname -r)

The initramfs-blahblah.img file is a replacement for the old initrd-blahblah.img file. So, we’re making a backup of the original initiam ramdisk image file for the running kernel and adding nouveau to its name so we know this is the initial ramdisk image that contains the nouveau driver (Fedora added the driver to the initial ramdisk so the graphical bootloader can take advantage of the NVidia chipset’s capabilities).

Then, running the dracut command creates a new initial ramdisk image for the running kernel. The dracut command replaces the mkinitrd that has been used traditionally. For more information about dracut check out the Fedora Project’s wiki page on dracut.

Finally, run the setsebool command RPMFusion’s page mentions:

setsebool -P allow_execstack on

If you’re like me, however, you probably have SELinux set to permissive because RPMFusion’s nonfree codec packages have already broken some SELinux stuff. Hopefully that will be fixed soon.

Then, reboot again into runlevel 5 and enjoy.

UPDATE: Read below!

Soon after getting kmod-nvidia installed, I noticed some weird issues in KDE. Whenever I would press ALT-F2 to run a command, the UI would freeze for about 10 seconds. I did some searching and found this was a reported bug. I’m guessing a forthcoming xorg-x11-server-* package update will include this, but in the meantime, I installed new xorg-x11-server-Xorg and xorg-x11-server-common packages from this 1.7.1-12 Koji build. Pressing ALT+F2 does not freeze the system anymore.

I’m almost a text-based e-mail purist. I used to use Mutt as my primary e-mail client application, but decided to go with a graphical client such as Mozilla Thunderbird or KMail so that I could at least effectively read HTML-formatted messages.

I’ve been happy with KMail. I’ve had it configured to prefer text-based e-mail and aside from the fact I don’t use my preferred text editor (vim) inside it, it’s been a good e-mail client. Now, my dad is a more typical e-mail user. While he probably doesn’t care that much about composing original HTML messages, he does receive a lot of them that he wants to forward onto other people that he feels may be interested. He’s on lots of political and family mailing lists that swap HTML messages complete with embedded images, etc.

He has been using Thunderbird at his home and KMail (an old version running on a Fedora Core 6 desktop) at his office. He mentioned to me that KMail runs noticeably faster on his work system than Thunderbird does on his home system. I suggested that we could standardize him on KMail and upgrade his office desktop to a more recent version of Fedora Linux.

Things got more interesting when Thunderbird recently got updated on his home system in a package update to version 3.0b4. The Smart Folders “feature” threw both of us for a loop. It combines multiple Inbox, Sent, and other IMAP folders into single virtual folders containing an aggregate of messages from each corresponding folder. I really have no idea who would want this feature. My parents each have their own e-mail accounts and I had Thunderbird configured so they could check mail for both accounts. The new version of Thunderbird combines both inboxes into one virtual “Smart” folder and subsequently confused the heck out of my father.

I figured out how to disable the “smart folder” behavior (View->Folders->All), but Thunderbird was still hiding other IMAP folders like Sent and Trash that my parents often need to access messages in.

So, KMail. KMail works great for almost all things, but my father noticed right away when he tried to forward an HTML message with embedded images that KMail wasn’t letting do what he was used to doing: Editing the forwarded message to remove the annoying gazillions of e-mail addresses the original message(s) were addressed to.

KMail has two methods of forwarding a message: First, you can forward a message as an attachment. This preserves everything about the original message, but KMail doesn’t let you edit anything within the attached message. Alternatively, you may forward a message “inline”. This lets you edit the message, but it only gives you the text portion of the message to edit and completely omits the HTML attachment altogether.

I did some research online to see if there was a way to get the desired functionality out of KMail, but it doesn’t look like it’s possible. If it does ever happen, it’s a couple versions out at least. It may never happen because it seems there are voices within the KMail community that feel KMail should never take on these types of features because it risks KMail becoming “another Outlook/Thundebird clone.”

Has anyone found other solutions to this problem for a Linux user?

The ffmpeg package that is available for Fedora 12 via the rpmfusion.org respository does not include faac support. This can be a problem when you want to create H.264 video content that incorporates the AAC (Advanced Audio Codec).

The most straightforward way I’ve found to rectify this situation is to build a new package from the source RPM.

First, download the source RPM using yumdownloader.

yumdownloader --source ffmpeg

This will download the .src.rpm file to the current directory. Install it using the rpm command. (This assumes you have a person RPM build environment set up. This blog post provides some good information on that.)

rpm -ivh ffmpeg-0.5-5.20091026svn.fc12.src.rpm

You probably want to indicate some sort of difference in the version numbering since this version is a modification of the upstream. Edit ~/rpm/SPECS/ffmpeg.spec and modify the Release: line by adding something to the end of it.

Release: 5.%{svn}svn%{?dist}_fozz

Now, you can try building the package with rpmbuild. Unless you’ve already installed all the development libraries and other dependencies ffmpeg relies on, you’ll get some dependency messages. Use yum to install those dependencies and then try building again.

rpmbuild -ba ~/rpm/SPEC/ffmpeg.spec --with faac

This will create RPM packages for you under ~/rpm/RPMS/. Use rpm to install the ffmpeg and ffmpeg-libs packages.

rpm -Uvh ~/rpm/RPMS/x86_64 ffmpeg-{libs-,}0.5-5.20091026svn.fc12_fozz.x86_64.rpm

The FFmpeg::Command Perl module is a convenient way to drive the ffmpeg command-line utility for converting multimedia files.

For work, I have developed some scripts that make heavy use of FFmpeg::Command. Yesterday, one of the other developers told me they need a conversion script to be able to merge separate video and audio streams into one file that contains both audio and video. The ffmpeg command-line utility can do this by accepting more than one input file. For example:

$ ffmpeg -i video.avi -i audio.wav -acodec copy -vcodec copy merged.avi

The FFmpeg::Command Perl module, however, assumed there can only be one input file. I made the necessary changes to the module code so that it would accept multiple input files, created a patch file, and sent it to the Module owner Gosuke Miyashita. This morning, I received e-mail from Gosuke thanking me for the patch and informing me that he has uploaded a new version (v0.12) of FFmpeg::Command to CPAN.

I love Perl and open source software!

I think I have finally, really arrived.

I’ve been doing contract work for a company in Provo that is launching a new website called YoManSports.com, which is in beta right now. At first glance, the site may appear to be a “YouTube for sports,” but it so much more than that. The concept is centered around video sharing, but includes familiar social networking elements you’d find on sites like Facebook or MySpace. In addition, there are several applications within the site that are sports-related — things like competition bracketing, scorecards, and groups. Perhaps the coolest feature that rounds out the list is the broadcast feature. This lets a person go to a sporting event with a video camera, even something as simple as a USB webcam, and set up a live web broadcast that anyone with a web browser can watch. The person managing the broadcast can mix prerecorded video, pictures, and even live video from other users into the broadcast. There’s even a news ticker for embedding clickable URL links into the broadcast. It’s pretty cool stuff.

Now, I said at the beginning of this post that I have arrived because we’ve been asked by management to blog regularly about the site and what we’re doing with it as part of our marketing plan. So, yeah, it’s cool to be able to do this and not be wasting my time at work.

My job has been designing and building the server architecture that sits behind the scenes and makes it all work. I was brought in late 2008 when the site was pretty much in a prototype stage. All the code was running on a single server and it really wasn’t designed to scale beyond that one server. So, one of the first things I did was figured out what we’d need to do split things like streaming video, web services, and database services onto their own dedicated servers.

After that, I went through and figured out how we were going to accomodate loads higher than we could with individual servers. In a nutshell: load balancing. That has now been implemented.

Another thing I’ve had a big hand in is offloaded encoding and conversion. The developers had created routines to do all the video encoding in PHP on the frontend of the website. Of course, doing video encoding on the same server Apache is running on can be detrimental to the experience of other website users. I developed a distributed encoding system that handles all the video conversion and encoding on a separate set of servers. I did it with Perl, of course.

I’m pleased with the technology being used on the site. I’m not a fan of PHP, but it’s doing the job well for frontend development. We’re making use of a lot of open source technology in dealing with videos. For example, all our transcoding is being done with the formidable FFMPEG software along with libraries like x264 and FAAC.

We’re leveraging Flash pretty heavily pretty heavily to make the site work so it’s fortunate that Flash support has nearly ceased being a problem for cross-platform compatibility. YoManSports.com works almost seamlessly across Windows, Mac OS, and Linux.

Watch this space for more info to come.

KDE 4.2 came down in the latest batch of updates for Fedora 10 last night. Resource settings were automagically migrated and it seems to have worked pretty well. Wikipedia has a decent summary of what's in KDE 4.2.

I’ve had an analogy on my mind for several days, churning, and I think I’d better blog it or else it’s going to devour me. Here it is:

“PHP application development is like bologna as Perl application development is like a good marinated London Broil.”

So, there you go. Now, let me explain. There are upsides and downsides to this.

Balogna requires nearly nothing to start eating. It’s as simple as opening the package, removing a slice of the processed meat product, and eating it. It’s just that easy. My five-year old does it all the time. Sometimes, he puts it on a plate and, using a butter knife, cuts the slice of balogna into little wedge-shaped pieces.

And, as a consumer, you don’t have to do much to get balogna to a point where you can consume it. It’s ready in its edible form at the supermarket. You go, buy it, take it home, (cut a slice into little wedge-shaped pieces), and eat it.

London Broil, on the other hand, isn’t available at the supermarket in sealed ready-to-eat packaging. You have to find a cut of top round beef, preferably one that has a nice amount of thickness (over 1-inch thick is ideal). In addition to the beef, you’re going to need to get some other ingredients for the marinade. When I was in college and wanted to impress a girl on a date, my sister suggested preparing a dinner that included London Broil. I don’t remember the exact recipe for the marinade now, but I remember there was worchestershire sauce, red cooking wine and series of different spices and salts. Looking online, there are many different recipes for such a marinade that include honey, garlic, chopped parsley, pepper, soy sauce, and more.

Once you get your ingredients home, mix the marinade and put it in a large plastic bag that zips shut. Place the raw beef into the bag with the marinade and put it in the refrigerator for up to 24 hours, turning it over once.

Cooking the marinated London Broil is a tricky p;ocess. You definitely broil or grill the meat, but this isn’t a hamburger. You need to heat it carefully, about six to eight inches over the flames. Some recipes actually call for broiling the meat to “rare” before you place it in the marinade. After the meat is cooked to the desired wellness, take it off the grill and begin cutting it in slices with your knife at a 45-degree angle to the meat.

Mmmmm mmmm. This is making my mouth water, just blogging about it!

London Broil, a magnificent meal that is sure to impress a date (or scare her away, as was my case.).

To bring this back to the programming languages, let me regurgitate a story of a recent experience I had.

A couple of weeks ago, a handful of systems I used to manage were compromised. These servers were running Fedora Core 6 and Fedora Core 5 which, of course, haven’t been supported by the Fedora community in at least a couple of years. The obvious response to a compromise was to install a actively supported Linux distribution on the hardware, make sure security issues are addressed, and then re-deploy the applications.

I was asked to assist in the process because of my knowledge of the systems.

Backing up critical data, installing a new OS, restoring data was pretty straightforward and easy. Next came the harder part: Getting all the applications configured and working again, just as they were supposed to.

Most of these applications were written in Perl. I went through each, one by one, taking note of the errors that occured when I first tried to run them. These errors invariably complained of missing CPAN modules — most of which were not available as packages available to install from a software repository affiliated with the Linux distribution.

This meant I had to go through, one by one, and build packages for each CPAN module that was a dependency for the applications. Many of these had their own sets of dependencies. The result: a couple of hours building packages to satisfy an interconnecting web of dependencies. In the end, everything worked.

It was then I decided to check on the one application that wasn’t written in Perl. This last application was written in PHP by some no-name programming team that the client paid to develop and host the software but when they didn’t have the chops to host and manage the application, the client took the application to someone who did have the necessary skills.

Guess what. The PHP application ran, out of the box. no unmet dependencies; No package installations neededl No fuss; Nothing.

At a recent PLUG meeting, I shared my experience with a friend who nodded in agreement. He too had his share of trouble navigating the waters of “dependency hell” trying to get a Perl application working.

Some Perl developers don’t understand this experience because they don’t use their OS’s packaging infrastructure to manage their Perl installation. Instead, they let Perl run loose, so to speak, and install necessary packages outside of a package management system. The upside: It’s fast. The downside: It’s risky and unmanagable.

My friend said, “Perl developers are just too smart.” I have to agree. Damian Conway, a Perl guru that travelled to Utah and spoke to a group of us about four years ago, likes to quote the late Arthur C. Clarke: “Any sufficiently advanced technology is indistinguishable from magic.” Conway prides himself on being the author of several CPAN modules that work “like magic.”

Conway’s not the only one. Many in the Perl community have developed extremely useful but complicated pieces of code and have graciously shared it with the open source community. Like any good open source community, others have built on what has been done and the result is software that has a deep root system of module dependencies.

Meanwhile, Perl has fallen out of favor as a language of choice for web application development, despite all the “magic” that exists within the Perl community. Why? If it’s so technologically superior, why aren’t the hordes of web developers using it?

The answer: Precisely because it is so technologically superior.

Newcomers to web development are drawn to the simplicity, straightforwardness, and relatively painless entry of developing applications using PHP. I can’t tell you how many PHP-based web applications I’ve looked at that are quite useful and powerful on the outside, but the code is... well, it’s boring. That’s not to say the programmers didn’t know what they were doing, they just didn’t really seem to know of any optimal ways of doing it. In the end, that’s okay, because the software works as it’s supposed to. A computer scientist will tell you, however, that this type of programming runs the risk of becoming unmanagable as it grows. The PHP community doesn’t seem to mind, though. They’ve had no problem “brute-forcing“ their way through most obstacles like this.

As a advocate of Perl, I’m left with a problem. My language of choice is failing at popularity contests and now I think I know why: It’s a pain in the ass to grasp in order to wield the magic.

What is it going to take to rectify this problem?! A Linux distribution that includes a great portion of the CPAN modules a programmer would ever need? That would certainly make things a lot easier.

What if the great minds of Perl put their current challenges aside for a few moments and tackled this challenge instead? Make the magic easier to obtain. Make using Perl much less frustrating for the uninitiated.

There are some that would say selling London Broil in a ready-to-eat package would be too hard or that it would be too expensive. I don’t know. I know you can get some very tasty prepared, marinated meats ready to slap on a plate. Sure, you pay more than you would for a slice of balogna, but isn’ it worth it?

I've been pretty happy with Fedora 10... until last night. That's when I hooked up my Palm Centro and tried to sync it with the kdepim apps.

It seems the usually-reliable kpilot has been replaced with a synchronization system based on opensync. I only wish it worked half as well as kpilot did.

So, I am at a crossroads, of sorts. I can either roll back to Fedora 8 and run KDE 3.5 where kpilot works nicely or I can get involved with the opensync.org folks and get the Palm synchronization code fixed/finished.

Fedora 10Fedora 10 was released 25 November (earlier this week), a mere six months after Fedora 9 made its debut. The Fedora community has been hard at work improving the distribution which is the descendant of Fedora Core and Red Hat Linux.


Looking over the release notes, it's obvious a lot of changes have been in the works. To name a few:

  • A new graphical boot system named Plymoth displays a simple, but effective boot progress screen that leaves old-school hackers like me feeling left out on the details.
  • PulseAudio has been rewritten to be "glitch-free." Makes you wonder why they didn't think of that in the first place. Having played (tried to play) Quake III with Fedora 9, I sincerely hope the experience under Fedora 10 is indeed, glitch-free.
  • As a KDE user, I am excited to see a more improved KDE4 platform. The kdepim package has been upgraded to 4.x so I hope to be enjoying a new kontact experience.
  • There's a new desktop alternative called LXDE. I'll have to try that out. It's a lightweight desktop which makes it perfect for those VNC sessions that come in handy from time to time.
  • My kids are already enjoying the new, improved versions of Extreme Tux Racer, Super Tux Kart, and other Fedora-bundled games.

But, unfortunately, I suck

About six weeks ago, I got involved with the Fedora Documentation Project with the aim of making a contribution to my prefered Linux distribution. After a handful of false-starts, I never did get acclimated to the contribution process, which involves learning about git (distributed version control), Fedora's docbook XML, the documentation project trac system, and Fedora's wiki-based stuff. I probably got about 80% through the process of learning how it all works but never got over the hump and actually started doing it.

My goal is to do that and be an active contributor for Fedora 11 and beyond. At the same time, I hope the documentation project leaders make an effort to decrease the "pain of entry" for those who may have lots to contribute but lack the experience working with the required tools.

Installing Fedora 10

I've installed Fedora 10 on four systems so far: three desktops and a laptop. On two of those, I did a network install by burning a bootable CD with the images/boot.iso image provided on the DVD.

One thing I noticed that is different doing network installs with Fedora 10 is that the network source can't just be a path to the ISO file. Instead, it needs to be the "exploded" ISO file directory structure. That's a little inconvenient as I don't like having to store the ISO and its exploded filesystem on my server resulting in about 9GB of used space instead of the normal 4.5GB.

Anaconda, the Fedora installer, has seen some subtle changes over the last couple of releases. One significant change is that the installer writes partition changes and formats filesystems much earlier in the installation process instead of waiting until package selection is finished.

Another change is the introduction of encrypted filesystem options. I haven't played with that yet.

One thing that is glaringly absent from Fedora 10 is the images/diskboot.img file which I have used in the past to create small bootable installation media using a USB flash drive.

It seems there is a way to do it still but you have to download a Live CD image (about 700MB) and use some commands to turn the Live CD ISO image into a bootable image you can write to a USB drive. There's lots of room for improvement here!

The systems I've installed on are as follows:

  • HP desktop, Intel P4 2.8Ghz, 512MB RAM, integrate i82865G graphics, 160GB SATA drive
  • Generic desktop, AMD Athlon 64 3800+, 1GB RAM, GeForce 6150LE integrated graphics, 80GB drive
  • Generic desktop, AMD Athlon XP 2500+, 1.3GB RAM, GeForce FX 5500 video, 2x 80GB drives
  • Dell Latitude D830N laptop, 2.6Ghz Core-Duo, 2GB RAM, NVidia Quadro graphics, 160GB SATA

Running Fedora 10

The only real problem I've encountered is on the HP desktop with games that required OpenGL accelerated graphics. Those games worked well on Fedora 8, but do not run accelerated on Fedora 10. Running glxinfo indicates direct rendering support is enabled, so I'm not sure what I need to do to get it working. If anyone has ideas, I'd love to hear them. Plus, my kids will enjoy playing SuperTuxKart on that computer again.

This weekend, I installed Scalix on a client's mail server. I had installed Scalix before, but it was a clean install with no old e-mail to migrate. This time, however, I had a lot of old e-mail to migrate.

Most of the users used Eudora for Windows. Others used the openwebmail web mail client. As a result, I had e-mail from three different sources to populate with:

  • Eudora locally-saved mailboxes (an almost mbox-format)
  • IMAP folders on the server, in mbox format
  • Standard mbox mailspool boxes (inboxes).

In preparation, I had read on a page on the Scalix site there were scripts to import mbox files into Scalix, but as it turned out, these scripts were for restoring a proprietary-to-Scalix backup mailbox format. Bleh.

The next best solution was to use an e-mail client that could import mbox mailboxes and move the messages to an IMAP folder. Thunderbird only imports a few formats on Windows and only imports Netscape Communicator on Linux. Bleh.

So, I tried kmail. It worked beautifully. At one point, I had three instances of kmail running in three different VNC sessions on the server funneling messages into Scalix. The only problem with kmail is that it required constant babysitting as I moved from mailbox to mailbox and user to user.

So, while kmail was cranking along on a large mailbox, I looked at CPAN to see what modules were available for dealing with IMAP and mbox file formats. It didn't take long before I had a script that would upload a user's messages. That worked great for the mbox mailspool files- there was one file per user.

Next were the IMAP/webmail mailboxes. I had copied them all into separate folders for each user, so to automate it, I needed a script that could go into each user folder and upload the messages from each of the mailboxes it found there. However, there were some mailboxes I didn't want to upload (e.g. Spam, Virus, Trash boxes). Others, I needed to rename to go into Scalix equivalents.

The following Perl script is what I came up with.


#!/usr/bin/perl

use Smart::Comments;
use Mail::IMAPClient;
use Mail::Box::Mbox;

my $users = [
    {   username   =>  'user1',
        password   =>  'password1', },
    {   username   =>  'user2',
        password   =>  'password2', }, ];

my $folder_translation = {
    'sent-mail'     =>  'Sent Items',
    'sentmail'      =>  'Sent Items',
    'saved-drafts'  =>  'Drafts',
    'saved-messages'=>  'Drafts',
};

my $ignore_folders = [ 
    'spam-mail', 
    'virus-mail', 
    'mail-trash', 
    'Junk',
    'Junk E-mail', 
    'Trash', ];

foreach my $user (@$users) { 
    if( -d "/export/imap/$user->{'username'}") {

        warn "Looking at $user->{'username'}\n";
        my $imap = Mail::IMAPClient->new(  
            Server => 'mail.example.com',
            User    => $user->{'username'} . '@example.com',
            Password=> $user->{'password'},) or 
            die "Could not log in as $user->{'username'}";

       
        opendir DH, "/export/imap/$user->{'username'}";
        my @files = readdir DH;

        FILELOOP:
        foreach my $file (@files) {
            if($file !~ m/^\./) {   # Skip hidden
                foreach my $ign (@$ignore_folders) {
                    if($file eq $ign) {
                        warn "Ignoring $file\n";
                        next FILELOOP;
                    }
                }
                my $translated_name = $file;
                foreach my $tr_key (keys %$folder_translation) {
                    if($tr_key eq $file) {
                        $translated_name = $folder_translation->{$tr_key};
                    }
                }

                # Ready to upload messages
                warn "Uploading $file to $translated_name\n";

                my $folder = Mail::Box::Mbox->new(
                    folder => '/export/imap/' . $user->{'username'} .  '/' . $file);

                my @folders = $imap->folders();
                if(! grep /^$translated_name$/, @folders) {
                    $imap->create($translated_name);
                }

                foreach my $msg ($folder->messages) { ### Uploading msgs |===[%]           |
                    my $uid = $imap->append($translated_name, $msg->string);
                    if(! $uid) {
                        warn "Could not append message\n";
                    }
                }

                $folder->close() or warn "Could not close Mbox connection $@\n";
            }
        }
        closedir DH;

        $imap->disconnect() or warn "Could not close IMAP connection $@\n";
    }
    else {
        next;
    }

}

About this Archive

This page is an archive of recent entries in the Open source software category.

Music is the previous category.

Perl programming is the next category.

Find recent content on the main index or look in the archives to find all content.