Work

I did some maintenance on our email server on Saturday (backup, chkdsk, defrag) just simple stuff, but somewhere inside the chkdsk things got mangled. I knew there were some file level problems on the machine, and there’d been a general stability problem for a while, so all of this was about due to happen. But what an annoyance, really. We had no inbound email for the entire weekend and Monday.

In the end, the problem was limited to our Xwall spam filter. It’s an incredible filter and I highly, highly recommend it. In this case a couple of it’s key files got re-arranged by the chkdsk, and it wasn’t forwarding emails onto Exchange after it checked them. It took me all weekend and yesterday to diagnose, I’m kinda kicking myself for it. All’s well that ends well I suppose, though.

In the process, I noticed the Appletalk protocol was still installed (from some old stuff we had some on the server, a couple of years back). I removed Appletalk, and then the even log started filling (every minute or two) with:

Event Type: Error
Event Source: atkctrs
Event Category: None
Event ID: 3001
Date:
Time:
User: N/A
Computer:
Description:
Unable to open ATK device for R access. Returning IO Status Block in Data.

Which was a real pain, and only frustrated the troubleshooting process. In the end one Kaushal Shah came to my rescue:

I am facing the same problem. The following Microsoft KB Article ID: Q257760 provides the solution.

It recommends dowloading Exctrlst.exe utility from the Microsoft Windows 2000 Resource Kit. Run this utility and uncheck all the performance counters for which you are getting a error.

Thanks buddy. In my case it was Windows 2003, but it didn’t make a difference; I ran the Exctrlst utility, dropped the performance logging on Appletalk (which, again, wasn’t installed anymore) and problem solved.

Half way through our holiday in NZ my laptop started puking out blue screens of death. It’s never done this before… but it started in earnest until the machine became pretty much unusable (over a period of 48 hours). So I spent some time in thought, regarding the fate of my 6 month old laptop that I love. After some diagnosis, and gentle coaxing, I was able to ascertain that my hard drive has bad sectors and they’re spreading like the plague. I don’t quite understand how that physical happens, but it’s related to a likely disk impact and scratching due to that. I travel this bad boy fairly heavily, so it’s not entirely surprising.

So my drive’s going bust and it’s a matter of time until it’s totally unusable. If I boot up the laptop and do absolutely nothing, it’s OK… I was able to run a full backup thankfully, so data’s fine. It’s when I tried to write anything that it would go all messed. To my great chagrin, that leaves me having to use Maija’s Macbook for the rest of my holiday. Conclusion: Mac’s are still useless for work. ;)

Anyway, we’re home in Finland now and I’m supposed to work later today, and I really don’t feel like using Maija’s computer anymore. Besides, she needs it for school. Enter Ubuntu Linux on a LiveCD. I’m a big fan of Linux, especially Ubuntu, but I felt I really had to take some time here to sing it’s praises.

Linux is a very customizable operating system. It can be re-arranged, rebuilt, and put together however you need it. These packages are called distributions, and Ubuntu is one of the most popular. I’m currently running Ubuntu 7.10, the Gutsy Gibbon. The beauty of a LiveCD is that the entire OS is packaged ready for use on a single CD-ROM. How does this help me? My hard drive is dead remember. I’m now running pretty much everything I need off a CD-ROM, and can continue working. Yes!

The Gutsy Gibbon is superb. Everything works out of the box: networking, wireless, sound, video, battery information, all my Dell keyboard buttons, etc. It comes with Firefox, Open Office, and Evolution. A good terminal server client, and various other media apps. Evolution is really quite a good mail client, and has support for Exchange. I still feel that Outlook is the best client out there, but Evolution is only getting better, while Outlook is only adding glut.
Graphically, the Gutsy Gibbon is lovely, possessing some of Vista’s Aero-Glass style effects. I’ve had a couple of problems with GParted (for partition disks), but aside from that’s it’s totally stable. Remember I’m running off a CD-ROM here, with everything cached into RAM, so it’s nowhere near an ideal setup. And even now it’s fast, it’s responsive.

I’m totally impressed. I’ve been using Linux for a few years, mostly server sided stuff, but this is the nicest thing I’ve seen on the desktop side yet. The only things I’ve run into that are lacking right now, are:
- good compatibility for Reason. It does run under Wine… but not so great
- multiple Exchange connections, in Evolution. In Outlook and Entourage you can open multiple inboxes, Evolution has yet to catch up.

You’ll notice those two things aren’t even Ubuntu core. Honestly, I think the time has come for Linux on the desktop. I encourage you to head over to ubuntu.com and grab a copy for yourself. Remember, it’s free.

(For the record, I’ve got a new hard drive on the way. I’m honestly torn as to whether to install Windows or stick with Linux. I don’t know what I want more. I’ll almost surely dual boot the system, as I need to use Reason and play old school videogames care of Home of the Underdogs… but beyond that? Don’t know.)

I was going to post this months ago, but I never got around to it. I needed it today, so I opened up the draft. May as well post it.

Locking down settings (such as proxy servers) in Firefox (and for multiple users). This article has seriously saved my bum a couple of times. Thanks man.

My task for the evening is to migrate our old Exchange server to a new one, with completely different hardware. This will be a long long post, and will be boring to those who know nothing about the words “Microsoft Exchange Server”.

Old server:
Dell Poweredge 4400
Windows Server 2003 SP 1
Intel Pentium III, 800 mhz
1 gb RAM, 133? Front Side Bus

New server:
Dell Poweredge 2850
Windows Server 2003 SP 1
Dual Intel Xeon, 3.6ghz
2 gb RAM, 800mhz Front Side Bus

Current time – 5:30pm

Our old mail server is plum tuckered out. We’ve decided it’s time for a new one, so we bought a fancy new server, and it’s been sitting in the rack for 3 weeks doing nothing. The migration task is difficult because we need the new server to do exactly the same thing as the old one, and hopefully without users noting anything has changed. Except for the extreme speed that will be coming their way once we’re working.

First step:
Backup all content on old mail server – I’m using the Windows Backup tool for this, which according to reports is better in Windows 2003 than it ever has been. So far it’s working, backing up everything on the server. I’m a little discouraged that it claims 2 days 6 hours are remaining, but I’m sure it will take much less than that. At this point things could have gone horribly wrong.

Thankfully, my boss called me and asked me how things were going. I told him of the 2 day time schedule, and explained that it was probably because we were backing up onto an external hard drive, and the old server only has USB 1.0. He said that was correct, and that I should try backing up to our network backup storage. Good idea says I. So I restarted the backup to the new location. Now I’m told 2 hours remaining. Woot woot woot.

… 1h 33m…

…30m…

Done. Next, go to our new Exchange server and restore the backup. When I started this I noticed the drive mappings weren’t the same on our new one as they were before, so using Disk Management I swapped the Data partition and CD-ROM drive letters. Now everything looks pretty good to go… so we check advanced settings. Make sure all files are replaced, even if more recent ones are on the server. And… go. 55 minutes remaining. In the time it took to write this section (about 5 minutes due to a brief interruption) it jumped down to 14 minutes. Yay!

…6m… At this point I figured it would be a good idea to shut down the old mail server. Just in case when the new one restarts there are IP and computer naming conflicts. “Planned shutdown: OK”

Restore complete. Restart computer. First boot:

Windows could not start because the following file is missing or corrupt:
\system32\ntoskrnl.exe

Shite. Rebooting from Windows Setup Disk and into Recovery Console… Must rebuild boot menu. Restart. Done that… didn’t help. Gave another more elementary error, so I’m going back into Recovery to try and fix the ntoskrnl specifically. Replaced the file, getting the same message as my manually created boot item.

Ok, I’ve rebooted again, and I’m doing the Autmated System Recovery this time. Maybe I should have started with this. This will (and is) installing a barebones Windows setup and will then ask for the location of the backup. I had a few problems with swapping floppy disks (due to requiring third party RAID drivers) but it’s actually quite intelligent. It’s now copying files.
Current concerns: a) My backup is stored in a network volume… that could be tricky. b) I don’t know if it will make the new partitions the same as the old ones. If so, that’s a bit undesirable. We have a much larger hard disk in the new server.
…28%…
Complete. Rebooting. The install process started, mostly as usual, and then asked for the backup media. Problem: It’s on the network. I can’t access the network. I am able to get into the System Applet, somehow though, so I’m going to try and install the drivers for the NIC, then join it to the domain.
Current time – 9:37pm

Supreme excitement!! I managed to intall the Network Drivers. If you click cancel enough on all the hardware found windows (remember, it’s only installed a very basic windows. You don’t have anything more than the install GUI) you’ll eventually get to the “Devices found but not installed” page. On that list were two Network Adapters. The install worked for the first, and then the second, and then I could connect to the Network Backup Device. Oh man I’m thrilled. So right now it’s restoring the backup. 21 minutes remaining.

Complete. Reboot. Boot successful! So, I’m in Windows now. It’s all the same as the old backup server… only it’s fast. And all the drivers have installed successfully. Only problem, the data partition doesn’t seem to have copied. I also can’t connect to anything on the network. Restart and boot into safe mode. Uninstall Network Adapters, reinstall. Reboot into Safe Mode with Network Support. Connectivity has returned, but I can’t open the backup file anymore (in order to restore the Data partition). Fingers crossed… rebooting into normal mode.

Normal boot successful. For some reason though when I open the backup file again nothing appears in the file list. Thankfully, while this was going on I copied the backup to my external hard drive, and now I tried using that copy instead. Success! There were a couple of new files in the directory the backup was stored, so I wonder if they were causing the backup to not work. Either way it’s now backing up the Data partition, 15 minutes remaining.
Current time – 10:20pm

Backup complete. The data partition is now in place. I rebooted. Low and behold… she’s alive!

That’s right, everything works! Outlook, web access, everything. There’ll be a bit of cleanup as there are some errors in the event log, but on the whole… I’m a happy camper. Thanks for reading. End time – 10:40pm