Just over one month ago I migrated our web site from Drupal to Wordpress. The migration went flawlessly and everything except the comments transferred without problem. The only content that didn’t migrate was comments from the old site. They were more difficult to migrate cross-CMS than the rest of the content and we decided the benefit did not justify the effort. Following the migration I tested everything (or so I thought), both from inside and outside our network. All indicators were green and everything worked as planned.

Then, a couple of days ago I looked at the site’s Google Analytics reports for the first time since the migration and I said “@#$&” and  “$%&^” and “%&$#?@!” too. And I meant every word of it. Why did I suddenly break out in grawlixes? Because I saw the traffic graph on the right. Our web site traffic fell off of a cliff and never recovered after the migration. According to Google Analytics our traffic was down 98.17% from our historical average.

How did that happen? It happened because I failed to check that all of the permalinks matched the old site. I had spot-checked some and they were the same, and so I assumed they would all be the same. That was an enormous mistake because Drupal and Wordpress generate permalinks from post titles in subtly different ways. For example, given a title containing the phrase “Ubuntu 9.04”, Drupal would generate a permalink containing “Ubuntu-9-04” and Wordpress would generate a permalink containing “Ubuntu-904”. This meant that a large number of our archives had new permalinks that were now one character different from the original. In fact, all of our most popular permalinks were affected. Bookmarks, inbound links and search results were all getting a 404 page, and not a very good one at that.

Much of this traffic is lost forever. After correcting most of the permalinks traffic is up to approximately 20% of our pre-migration levels. I expect more of it to return as search engine crawlers clean up after the mess I caused. But it won’t all return and it will take months to rebuild to pre-migration levels. If our business depended on web site traffic, I would likely be updating my resumé now instead of writing this.

Lessons Learned

I learned these lessons the hard way so you don’t have to.

• Compare the permalinks from the old and the new CMS. All of them.
• Look at your web stats reports regularly.
• Make sure you have a helpful 404 page. One that says “404: Page Not Found” and nothing else is not helpful.
• Tag and categorize your content. The 404 page for the Wordpress theme I’m using uses this information to help a visitor landing on the 404 page navigate to somewhere more useful.
• Get a real-time report of 404 errors. There are several Wordpress plugins that will help. I used the 404 Notifier plugin by Alex King. That plugin sends an email anytime a 404 error occurs, but there are other plugins available that serve a similar purpose. If I had this kind of real-time report during the migration, the problem would have been identified and corrected much sooner and the traffic impact would have been minimized.
• Use a plugin that tries to locate the missing content. I’m using Smart 404 for Wordpress by Michael Tyson. Smart 404 tries to find the missing page and redirects to the correct page. If it can’t locate the correct page then it tries send the visitor to a relevant category or tag. There are similar plugins that take other approaches, such as using a site map and a local Google search to locate either the missing content or relevant content.

The solution I found was faad and faac – Freeware Advanced Audio Decoder/Coder. It will decode unencrypted AAC/MP4 files and will happily ignore and correct the munged frame duration in the MP4 wrapper. I recovered a damaged track like this:

faad "09 This Devil's Workday.m4a" -o "09 This Devil's Workday.wav"

mv "09 This Devil's Workday.m4a" "09 This Devil's Workday.m4a.bad"

faac -w --artist "Modest Mouse" --title "This Devil's Workday" --album "Good News for People Who Love Bad News" --track 9 --disc 1 --year 2004 "09 This Devil's Workday.wav"

As faad was decoding the original track it showed that the MP4 wrapper’s duration was, but it corrected as it decoded it.

MP4 seems to have incorrect frame duration, using values from AAC data.
Decoding 09 This Devil's Workday.m4a took: 1.62 sec. 85.94x real-time.

When faad/faac were done I was able to re-import the track into iTunes and play it successfully.

Caveats

I lost some of the iTunes metadata from the original track, mainly the cover art. iTunes automatically added the cover art when I re-imported it, but that only works for music in the iTunes library. In those cases, faac has an option to import cover art from an external image file. You can do the same thing with iTunes as well.

If you have a lot of tracks to repair, adding the metadata manually on the command line is tedious. However, it could be automated with a simple script that captures and processes the output of faad since it displays all the track metadata as it is processing.

I put a band-aid over this problem by adding /dev/pty to /etc/fstab. After revisiting this bug report I found that the source of the problem was that I was missing a symlink in /etc/rcS.d. If you are having the same problem first check that
/etc/init.d/mountdevsubfs.sh exists and then run the following commands:

cd /etc/rcS.d/

sudo ln -s ../init.d/mountdevsubfs.sh S11mountdevsubfs.sh

However, there is no explanation for why this symlink was missing in the first place.

No RAID

This turned out to be an extremely easy problem to fix. While fixing the symlink problem above I noticed that init scripts weren't ordered correctly. The init scripts for disk mounting were happening too early, before the mdadm module and services had fully loaded. That's why my RAID volume wasn't mounting on startup but I could mount it manually after logging in. I corrected the order of the init scripts and now the RAID volume auto-mounts when booting.

No Terminal

When I tried to launch GNOME Terminal from the menu I got this message:

There was an error creating the child process for this terminal

Not good. The first thing I tried was to start Update Manager and apply any patches. Update Manager showed me a list of patches but when I tried to install the updates it failed with the message:

Error failed to fork pty

At this point I wasn't getting warm, fuzzy feelings. Googling for the error messages turned up several Ubuntu bug reports for both problems. Many commenters reported that adding devpts to the fstab resolved the problem for them. So I tried their suggestion and it solved the problem for me as well. To apply this fix you need to add the following line to /etc/fstab:

devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0

To edit /etc/fstab press ALT+F2 to bring up the Run Application dialog. Then type in either”

gksudo gedit /etc/fstab

or:

gksudo gedit /etc/fstab

After adding the entry for devpts you need to mount it. You can either reboot or you press CTRL+ALT+F1 to temporarily switch to console mode. Once in console mode, login as root and run:

mount -a

exit

After exiting console mode you should now be able to successfully launch GNOME Terminal from the menu. It worked for me anyway.

NOTE: After rebooting, GNOME Terminal would not work again. After running mount -a from console mode it started working. So the problem is only partially solved if I have to manually mount the device after each boot.

No RAID

After the upgrade my RAID volume would not mount. When I tried to mount it I got this error message:

mount: special device /dev/md0 does not exist

I ran sudo mdadm --detail --scan and it showed the RAID array, but it didn't match the entries in either /etc/mdadm/mdadm.conf or /etc/fstab. It turned out that the RAID device had changed from /dev/md0 to /dev/md/d0 in the upgrade. Fixing the problem required two steps:

• Replace the ARRAY entry in /etc/mdadm/mdadm.conf with the output from sudo mdadm --detail --scan.

  NOTE: If the output contains the phrase metadata=00.90, then delete that phrase. mdadm doesn't like metadata format 00.90 and gave me an error message:

  mdadm: metadata format 00.90 unknown, ignored.

• Change the entry in /etc/fstab to the new device, i.e. from /dev/md0 to /dev/md/d0.

NOTE: The RAID volume does not auto-mount after rebooting. So this problem is also only partially solved.

No Sound

When anything tries to play sound I just get a crackling sound from the speakers. I'm still working on this problem…

Update: Sound problem fixed.

Final Update: All problems fixed.