Recently in Work Category

The FFmpeg::Command Perl module is a convenient way to drive the ffmpeg command-line utility for converting multimedia files.

For work, I have developed some scripts that make heavy use of FFmpeg::Command. Yesterday, one of the other developers told me they need a conversion script to be able to merge separate video and audio streams into one file that contains both audio and video. The ffmpeg command-line utility can do this by accepting more than one input file. For example:

$ ffmpeg -i video.avi -i audio.wav -acodec copy -vcodec copy merged.avi

The FFmpeg::Command Perl module, however, assumed there can only be one input file. I made the necessary changes to the module code so that it would accept multiple input files, created a patch file, and sent it to the Module owner Gosuke Miyashita. This morning, I received e-mail from Gosuke thanking me for the patch and informing me that he has uploaded a new version (v0.12) of FFmpeg::Command to CPAN.

I love Perl and open source software!

I think I have finally, really arrived.

I’ve been doing contract work for a company in Provo that is launching a new website called YoManSports.com, which is in beta right now. At first glance, the site may appear to be a “YouTube for sports,” but it so much more than that. The concept is centered around video sharing, but includes familiar social networking elements you’d find on sites like Facebook or MySpace. In addition, there are several applications within the site that are sports-related — things like competition bracketing, scorecards, and groups. Perhaps the coolest feature that rounds out the list is the broadcast feature. This lets a person go to a sporting event with a video camera, even something as simple as a USB webcam, and set up a live web broadcast that anyone with a web browser can watch. The person managing the broadcast can mix prerecorded video, pictures, and even live video from other users into the broadcast. There’s even a news ticker for embedding clickable URL links into the broadcast. It’s pretty cool stuff.

Now, I said at the beginning of this post that I have arrived because we’ve been asked by management to blog regularly about the site and what we’re doing with it as part of our marketing plan. So, yeah, it’s cool to be able to do this and not be wasting my time at work.

My job has been designing and building the server architecture that sits behind the scenes and makes it all work. I was brought in late 2008 when the site was pretty much in a prototype stage. All the code was running on a single server and it really wasn’t designed to scale beyond that one server. So, one of the first things I did was figured out what we’d need to do split things like streaming video, web services, and database services onto their own dedicated servers.

After that, I went through and figured out how we were going to accomodate loads higher than we could with individual servers. In a nutshell: load balancing. That has now been implemented.

Another thing I’ve had a big hand in is offloaded encoding and conversion. The developers had created routines to do all the video encoding in PHP on the frontend of the website. Of course, doing video encoding on the same server Apache is running on can be detrimental to the experience of other website users. I developed a distributed encoding system that handles all the video conversion and encoding on a separate set of servers. I did it with Perl, of course.

I’m pleased with the technology being used on the site. I’m not a fan of PHP, but it’s doing the job well for frontend development. We’re making use of a lot of open source technology in dealing with videos. For example, all our transcoding is being done with the formidable FFMPEG software along with libraries like x264 and FAAC.

We’re leveraging Flash pretty heavily pretty heavily to make the site work so it’s fortunate that Flash support has nearly ceased being a problem for cross-platform compatibility. YoManSports.com works almost seamlessly across Windows, Mac OS, and Linux.

Watch this space for more info to come.

As many casual readers of the Fozzolog may know, I’m now working for a new company called KnowledgeBlue. Things are done differently at KnowledgeBlue and it’s not unexpeced that one company would do things differently than another, but the way things are done at KnowledgeBlue has been a bit of a shock to my system. I think my Iodynamics colleagues are experiencing similar feelings.

Robb and Chris are veterans of the corporate IT world and I don’t mean the IT department of your standard local business but the IT world of a large corporation listed on the Fortune 500 list. As a result, they bring an amount of discipline and regimen to workflow that I’m not acustomed to working with.

That being said, I’ve been learning a lot and enjoying it immensely. (I’m not just saying that to be a brown-noser.) While the regimented workplace is, without question, overkill for the size of company KnowledgeBlue is right now, the company will be growing significantly in the coming months. Without these policies and procedures in place, it would be easy for things to get out of control.

In all honesty, the structure and rigidity has been frustrating and the amount of work and responsibility piling up on my desk has been putting my confidence through its paces (pacing through the doldrums, perhaps?) That being said, I haven’t been challenged like this in a long time and despite my confidence scraping the bottom of the proverbial barrel, those I’m working for are supporting me and expressing confidence in me. So, there’s that.

Project planning

One thing I’ve been requested to do is do project and resource planning for some of the work that is upcoming. In the “corporate world“ the de facto computer application for such a task would be Microsoft Project. Anyone who knows me knows... that ain’t gonna happen. Fortunately, there are many alternatives to choose from in the immense software catalog of the Open Source universe.

Here’s a preliminary rundown of what I’ve come across.

Kplato was one of the first applications I looked into and I actually used it for a few days before moving on. It does a fair job of generating WBS (Work breakdown structure) codes and tracking who is assigned to what tasks. The one thing that kind of became a dealbreaker for me is that kplato doesn’t seem to be able to associate prerequisite task information. That is, there doesn’t seem to be an easy way to say “Task A must be finished before Task B can begin.” You can only specify task starting and ending constraints by date.

Planner, formerly known as MrProject, is a project management app that has been around for a while in various incarnations. Planner has most of the features I’m looking for and, unlike kplato, does support the concept of “predecessor tasks.” Additionally, Planner features the ability to import data from Microsoft Project XML format files. There is a port of Planner for Windows also available. One thing that bugs me about Planner is that it’s a GNOME application and suffers from the UI braindeadness that accompanies most GNOME applications. Otherwise, it’s probably one of the most capable of the applications I’ve looked at.

TaskJuggler is a bizarre entry to this field of applications. It completely deviates from the project management software genre. Users are required to enter information about projects, tasks, resources, etc. using a programming language-like syntax into a text file which is then compiled by the application. The authors of TaskJuggler claim their program is leaps and bounds better than “traditional” project management software. I’m a bit reluctant to go down that road. I should definitely come back to TaskJuggler at some point. For now, I’m sticking with the traditional applications.

OpenProj is the first Java-based desktop app of the mix I’ve looked at. It features nice import/export compatibiliy with Microsoft Project and its cross-platform nature, thanks to Java, also makes it attractive. It also boasts task predecessor support. It’s not very easy to use compared to the others I’ve looked at and it runs slow. It does have a lot of screens and reports, though.

GanttProject is another Java desktop application I looked at. It offers good import/export capabilities for people who are dealing with Microsoft Project data. It doesn’t offer quite as many views as OpenProj, but tries to make up for this with lots of export options (HTML. PDF, CSV, PNG, and JPEG). GanttProject also supports the concept of prerequisite or predecessor tasks. One disappointment, however, is that GanttProject doesn’t seem to let you get any more precise on the length of effort of a task beyond the number of days. For example, you can’t express that a task is only going to take 2 hours to complete. It also doesn’t handle some errors very well.

I’d love to hear about other applications that I haven’t listed above.

Those who attended the dinner and/or first keynotes from tonight’s Utah Open Source Conference session may have seen a press release on display at KnowledgeBlue’s exhibition area. It announces the merger of Iodynamics into KnowledgeBlue.

So, no more secrets! I’m a KnowledgeBlue employee now as well as Mike, Adam, and Thom - my chums from Iodynamics.

KnowledgeBlue is based in Salt Lake City and is in the business of (my words here) gluing together open source software packages into systems that offer great value to businesses.

Another thing KnowledgeBlue is letting people know at UTOSC is that we’re hiring. If you’re a can-do person with a firm understanding and appreciation of Open Source software, check out the jobs available on the KnowledgeBlue careers page.

"There are no graduates. The learning never stops." -- Rush Limbaugh says that every once in a while when identifying the Rush Limbaugh Institute for Advanced Conservatives Studies.

I like the mantra, today it applies to me, and I think it should apply to everyone in the open source community.

We recently got a call from a guy with a company out of Utah County who needed some work done on a Linux server. The first meeting took a couple months to set up, but it finally happened about a month ago. Adam and I checked it out. He had a Dell rackmount box that had suffered some hardware failure of some kind. It was a pretty important server, so they hurried and built a new server on other hardware.

This led to them contacting us. Now that the original hardware was fixed, our client wanted us to document everything that was installed and configured on the rebuilt system and duplicate it on the original system -- fixing things that weren't done quite right along the way.

We did all that and delivered a 26 page set of documentation on the system.

Our client then requested that we do two more things. First, he wanted one machine to act as a failover slave to the other. Second, he wanted to do implement a single-signon scenario so that users of the Linux system could use their Windows network usernames and passwords to log into the Linux system via SSH.

I've set up a couple High-Availability Linux clusters before, so I wasn't worried about that, but I hadn't really investigated the single-signon thing before - especially not with Windows as the authentication source.

Turns out the single-signon thing wasn't that difficult. It comes down to a handful of steps:

1. Edit /etc/nsswitch.conf and add winbind after files for the passwd and group entries.
2. Configure /etc/krb5.conf with the right Kerberos realm names and server names.
3. Configure /etc/samba/smb.conf with a few winbind parameters like what is shown here.
4. Join the domain with net rpc join -S PDC -U Administrator.
5. Start the winbind service and restart Samba.
6. Edit login and sshd in /etc/pam.d and add lines like auth sufficient /lib/security/pam_winbind.so which tells PAM to use winbind as a source of credential validation.
7. Edit /etc/pam.d/system-auth and add a line session required /lib/security/$ISA/pam_mkhomedir.so skel=/etc/skel umask=0022 so that PAM will create a home directory (specified by the template homedir directive in the Samba configuration file) when a user successfully logs in for the first time.

And that's pretty much it.

Tonight was my final visit for these projects. We did it after hours so we could take the servers down for IP address changes and what-not. Two to three hours later, I had two servers using heartbeat failover and accepting Windows username and password logins. Very nice.

Google had their Summer of Code, I just had my...

Weekend... Of... CODE

(Say it like The Muppet Show's "Pigs In Space.")

It's been kind of nice, really. First of all, I enhanced the way gig images are viewed over at the Sons Of Nothing site. Instead of images opening up in a new browser window when you clicked on a thumbnail, the images will open up "inside" the site along with some handy "next image" and "previous image" navigation.

It wasn't hard. Just create a new template for the full-size image page and give it some meta data about the next and previous images in the set. Done.

Perl, mod_perl, and Template Toolkit just rock... like Sons Of Nothing rocks... but different.

For example, check out this recently-posted rare glimpse of me from our September jaunt to Steamboat Springs, Colorado.

Secondly, Iodynamics is currently involved in a project for a client that involves gluing a bunch of facilities together within one website including auctions, weblogs, news articles, file downloads, and more. Call in the next 20 minutes and you'll also get this handy dandy nosehair trimmer.

Anyway, the original project specification had us doing everything in PHP - not our first choice for anything. We weren't too worried about spending too much time writing PHP code because our plan was to use "off-the-shelf" components or applications and glue them together into one seamless website.

It never quite works that way, does it?

The "RSS feed" we were supposed to be using for the news articles turned out to be an XML format which borrowed haphazardly from RSS, but is completely different, thereby requiring some ground-up parser development.

The file download repository is specific enough, we hadn't been able to find anything that would do the trick admirably enough.

Oh, did I mention the client was seriously hoping to have it all done last week?

Fortunately, the client succumbed and said, "Go ahead and do it in Perl." Whatever it takes, just get it done quick. Man, after searching for something akin to CPAN's XML::Simple for PHP, it was so refreshing to be able to just use versatile Perl modules like XML::Simple, LWP::UserAgent, DBI, and others.

I was able to crank out all kinds of magic to do the news articles and the file download repository in an amazingly small amount of code and still maintain a strict separation of concerns (i.e. MVC) -- something that you have to work really hard to do when programming with PHP.

Server upgrades haven't been so smooth lately. We've been upgrading the Iodynamics core servers to Fedora Core 4, but it just seems like every upgrade we do something goes wrong.

Today, for example, Adam and I are upgrading what is, perhaps, the most essential server for the company: castro. I decided we should upgrade some hardware while we're at it and maybe that was a bad move on my part. Maybe we should have stuck with the hardware that had been working fine.

It's raining outside.

Anyway, we threw in a new motherboard and CPU and started the reinstall process with FC4. As soon as the drive formatting was complete and the install was about to start... Kernel panic. It happened every time.

The panic message said something about journalling. That indicates to me there's a problem with the kernel's interaction with the ext3 filesystems on the hard disks. Very strange considering the hard disks have been running fine.

Adam and I thought maybe the problem was due to bad memory (which was also odd considering the memory was also in the previous motherboard). We ran a memory test regardless. No errors.

The CPU temperature was pretty high: 85° C. The fan was seated well and turning when the system was turned on. The heatsink was getting real hot. Very weird.

I just sent Adam to a local computer store to get two new hard drives, more RAM, some thermal goo, and possibly a new fan. We'll see how it turns out.

(Time passes.)

Adam returned. We attached the new hard drives (SATA drives to replace the aging PATA drives) and the motherboard's onboard SATA controller BIOS won't POST. We tried EVERYTHING to get that SATA BIOS to POST and it wouldn't. We even used a USB keychain drive to update the BIOS on the motherboard, but that didn't seem to affect anything (i.e. it was already running the latest and greatest BIOS.)

So. Adam set out again... this time, to our office, to get yet-another motherboard -- one that we know works because Adam's been using it in his workstation at the office.

It's been nearly five hours since we started. Hopefully, this will be the last hurdle.

I'm getting pretty bad at keeping up to date this last month or so.

There really is not any exciting news to mention, I guess.

At Iodynamics, we found out a big client based in California that had given us the go-ahead on a good-sized web development project decided at a somewhat last minute to withdraw their support for Iodynamics doing the project and instead work with a local company. That threw a big wrench into our revenue projections and it got me a bit depressed. We talked about it and decided it wasn't going to mean layoffs or anything like that, although it's going to be tight for the next couple of months.

To make matters a little more annoying, we had already developed a separate website for this client and have been hosting it. It is, of course, a mod_perl-based site using a PostgreSQL database and Template Toolkit for presentation. The client said they would be taking that site "in-house" and one of their engineers would be in touch with us on moving the site to a server on their network.

Their engineer called and announced he wanted to move the site to a Windows server running IIS. It took me a while to clean my jaw up off the floor! He also asked us about running it on Windows with Apache, PostgreSQL, and mod_perl. I admitted it was possible, but I didn't recommend it. That's mostly because I could tell they had no expertise in setting up these software on Windows and I certainly never had any reason to do it.

In the end, we uploaded all our code to them and they'll take care of the porting. (Good luck!)

What a head-scratcher.