Today, Thom asked me if I was still “busybusybusy.” I said, “Yeah, but it’s settled down into a routine.”

And that’s a good thing, I think.

Today was a challenge, though, and think I met it head-on. This morning I went to work for a company I’m doing some contract programming for. I worked onsite there until 1 in the afternoon. Then I got some lunch and went to another client’s office until 4:30. Things were going pretty smooth all day.

Christine called during the day and said it looked very likely she’d be pulling a late night again at Sorenson Media because they were going to roll out some new software to some servers and they had to get it all tested and then do the deployment at 1 a.m. I asked her if she was going to come home for a while and then go back and she said she was just going to stay there, but she asked me to come see her and bring the kids.

So, I went home and got the kids. It was around 6 p.m. so traffic was a bit on the heavy side. It didn’t help we hit almost every red light possible on the way over. While en route, I got a call from a client who said they were having server issues. I called one of their programmers to see if he could get the server issues resolved. He restarted some services on a box, but that didn’t really help. One of the load-balanced servers appeared to be offline- so we either needed to make sure it was offline for real (i.e. so the load balancer couldn’t get a heartbeat off it) or reconfigure the load balancer so that it wouldn’t try to include the defective server in the load rotation. So, either I had to log into the remote power controller and power off the server or I had to use the Java-based SNMP software to remotely reconfigure the switch. Either way, I needed an Internet connection.

I had promised Christine I’d go get dinner at Subway for everyone, so I did that and when I got back to Sorenson Media, she took the kids into the breakroom while I tried to quickly resolve the situation with the servers by plugging my laptop into a network switch on her desk. Getting a valid IP configuration was my first challenge, but I managed to get something that would work. I disabled the server in the load balancer, but the other server was still having problems and the websites were not coming up.

I determined I needed to head to the colocation facility where the server farm was located and do some hands-on work. This presented a problem because Christine was expected to stay at work until 2 a.m. or later and I was expected to take care of the kids. So, I called my mom. She reluctantly agreed to meet me at our house and take care of getting the kids in bed.

I told Christine what I had arranged and she said not to to do that — she’d go home with the kids.

So I called and managed to catch my mom before she had left for our house and told her not to bother. Christine and I traded cars and I drove to the colocation facility.

There were a couple of problems on that server and I don’t know if they were related or not. One problem was that an Apache access log file had grown to 2 gigabytes in size and Apache didn’t like that. It was spitting error messages into the error log every second. That didn’t explain the error messages that came up when you tried to load the websites, though. That appeared to be caused by a MySQL server that was refusing connections to the server because of "multiple connection errors."

So, I did a flush-hosts on the database server and moved the too-big log file out of the way and restarted things. Everything went back to “condition green.”

I made it home by 10 or so. Christine called back into work and offered to come back in. They said they had more than enough people available to help. I hope they weren’t just saying that.

In conlusion of all this, I have to say I’m pretty impressed with myself. I think I handled things really well and I kept myself relatively cool. A few years ago I would have either started yelling at people for no apparent reason or I would have just started crying or something. Heh heh. No, I think I did a good job of staying cool in the face of advertsity and meeting the challenge head on. I could have been bitter that few people seemed willing to help me, but I think the point was that this was my responsibility.