Except for being greeted by the “Partial Upgrade” dialog in Update Manager the upgrade went as smoothly as can be expected for an in-place upgrade. I had Pidgin talking to Microsoft Office Communications Server within a few minutes of completing the upgrade. Mission accomplished. Then I needed to connect to Microsoft PPTP VPN at a clients’ site and I started to experience Ubuntu upgrade woes. The VPN connection was failing with the following error:

I spent quite a bit of time trying to fix that problem with no success. I couldn’t find any helpful errors in the system logs. The PPTP client was logging the following when I tried to connect:

pppd[18079]: Plugin /usr/lib/pppd/2.4.5//nm-pptp-pppd-plugin.so loaded.
pppd[18079]: pppd 2.4.5 started by root, uid 0
pppd[18079]: Using interface ppp0
pppd[18079]: Connect: ppp0 < --> /dev/pts/2
pppd[18079]: CHAP authentication succeeded
pppd[18079]: MPPE 40-bit stateless compression enabled
pppd[18079]: local  IP address 172.30.8.180
pppd[18079]: remote IP address 172.30.8.61
pppd[18079]: primary   DNS address 172.30.3.22
pppd[18079]: secondary DNS address 172.30.3.23
pppd[18079]: Terminating on signal 15
pppd[18079]: Connect time 0.7 minutes.
pppd[18079]: Sent 0 bytes, received 0 bytes.
pppd[18079]: Child process /usr/sbin/pptp vpn.********.com --nolaunchpppd --logstring nm-pptp-service-18077 (pid 18081) terminated with signal 15
pppd[18079]: Connection terminated.
pppd[18079]: Exit.

Not much info there. The NetworkManager log offered a little bit more info:

NetworkManager: <info>  Starting VPN service 'org.freedesktop.NetworkManager.pptp'...
NetworkManager: <info>  VPN service 'org.freedesktop.NetworkManager.pptp' started (org.freedesktop.NetworkManager.pptp), PID 15184
NetworkManager: <info>  VPN service 'org.freedesktop.NetworkManager.pptp' just appeared, activating connections
NetworkManager: <info>  VPN plugin state changed: 1
NetworkManager: <info>  VPN plugin state changed: 3
NetworkManager: <info>  VPN connection 'Corporate VPN' (Connect) reply received.
NetworkManager:    SCPlugin-Ifupdown: devices added (path: /sys/devices/virtual/net/ppp0, iface: ppp0)
NetworkManager:    SCPlugin-Ifupdown: device added (path: /sys/devices/virtual/net/ppp0, iface: ppp0): no ifupdown configuration found.
NetworkManager: <info>  VPN connection 'Corporate VPN' (IP Config Get) timeout exceeded.
NetworkManager: <info>  Policy set 'Auto eth0' (eth0) as default for routing and DNS.
NetworkManager:    SCPlugin-Ifupdown: devices removed (path: /sys/devices/virtual/net/ppp0, iface: ppp0)
NetworkManager: <debug> [1283289056.002038] ensure_killed(): waiting for vpn service pid 15184 to exit
NetworkManager: <debug> [1283289056.002152] ensure_killed(): vpn service pid 15184 cleaned up

The VPN connection was timing out. A Google search for that timeout message turned up lots of problems, but few solutions and none that worked for me.

While I searched for a working solution to my problem I launched Rhythmbox to play some “thinking music”. Rhythmbox loaded and then died. I started it again and it died again. More upgrade woes. I looked in the system log and found this message:

rhythmbox[29735]: segfault at 0 ip  013e97e2 sp b43fead8 error 6 in libnss_wins.so.2[13a8000+253000]

Rhythmbox was getting a SIGSEGV and dying shortly after launching every time. A Google search for that message turned up lots of other users experiencing the same problem. A few commenters reported that the issue was related to winbind. Several suggested the problem was the order of the hosts in /etc/nsswitch.conf and that moving the wins entry to the end of the hosts line would fix the problem. I edited /etc/nsswitch.conf and changed this line:

hosts:          files wins mdns4_minimal [NOTFOUND=return] dns mdns4

To this:

hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4 wins

Then I launched Rhythmbox and it did not segfault. Problem solved.¹

Now that I had music again I returned to working on the VPN problem. I tried connecting the VPN again and … it worked! Sonofa! The change I made to /etc/nsswitch.conf to fix Rhythmbox also fixed my Microsoft VPN problems too. Another win for serendipity!

¹Moving wins after dns on the hosts line in /etc/nsswitch.conf effectively disables winbind and this may cause problems with Samba. It didn’t cause any problems for me, but your mileage may vary.

To save you time I’ll give you the solution first then describe how I found it: Turn off Unix extensions in your Samba server by adding the following line to smb.conf in the global settings block and then restart Samba:

unix extensions = no
 

You might also need to unmount and re-mount your Samba volumes from OS X after you make this change.
 

After installing the OS X Snow Leopard 10.6.3 update I found that I could no longer write to Samba (SMB) volumes shared from my Linux server (running Ubuntu 9.10 Karmic Koala) that I had mounted on OS X. Whenever I tried to copy a file from OS X to the mounted Samba drive I got the error message:

The operation can’t be completed because you don’t have permission to access some of the items.
 

Apparently a lot of other people are having the problem as well:

• 10.6.3 Update Breaking SMB Mounted Volume Write Access for Many
• Topic : After update to 10.6.3 copy to a Samba share fails
• Topic : No rights to write on SMB mounted volume after updating on 10.6.3

I couldn’t find the solution anywhere on Google, and I spent hours today searching. I finally found the solution serendipitously while trying to fix another Samba issue. While scouring the system and Samba logs on my Linux server to try to find a clue to the first problem I found a bunch of warnings like the following repeated over and over in the Samba log:

[2010/04/05 00:08:37, 0] param/loadparm.c:9783(widelinks_warning)
Share 'documents' has wide links and unix extensions enabled. These parameters are incompatible. Wide links will be disabled for this share.
 

This warning started logging on March 24th, the same day that Ubuntu announced and patched a vulnerability related to the Samba wide links option: Ubuntu Security Notice USN-918-1. According to Using Samba by Robert Eckstein, David Collier-Brown and Peter Kelly you should not turn off the wide links option for performance reasons. (See Appendix B: Samba Performance Tuning.) Since I use Samba for just about everything I decided that performance was more important than Unix extensions so I added the following to my smb.conf in the global settings block and restarted Samba:

unix extensions = no
 

To my amazement and delight, this had the welcome side effect of solving the Samba write access issue I was having with OS X. To verify that this is really what solved the problem I re-enabled Unix extensions and restarted Samba and the write access problem returned. Then I turned Unix extensions off again and restarted Samba and I could write to all of the Samba mounts again from OS X. So if you are having problems writing to Samba mounts from OS X ask your system administrator to turn off Unix extensions in the server’s Samba configuration.
 

I’ve had emails and comments that point out that there are advantages to having Unix extensions on. That’s true and this should be considered a workaround rather than a fix. When (if?) Apple fixes the underlying problem you should turn Unix extensions back on in your Samba server.

Just over one month ago I migrated our web site from Drupal to WordPress. The migration went flawlessly and everything except the comments transferred without problem. The only content that didn’t migrate was comments from the old site. They were more difficult to migrate cross-CMS than the rest of the content and we decided the benefit did not justify the effort. Following the migration I tested everything (or so I thought), both from inside and outside our network. All indicators were green and everything worked as planned.

Then, a couple of days ago I looked at the site’s Google Analytics reports for the first time since the migration and I said “@#$&” and “$%&^” and “%&$#?@!” too. And I meant every word of it. Why did I suddenly break out in grawlixes? Because I saw the traffic graph on the right. Our web site traffic fell off of a cliff and never recovered after the migration. According to Google Analytics our traffic was down 98.17% from our historical average.

How did that happen? It happened because I failed to check that all of the permalinks matched the old site. I had spot-checked some and they were the same, and so I assumed they would all be the same. That was an enormous mistake because Drupal and WordPress generate permalinks from post titles in subtly different ways. For example, given a title containing the phrase “Ubuntu 9.04”, Drupal would generate a permalink containing “Ubuntu-9-04” and WordPress would generate a permalink containing “Ubuntu-904”. This meant that a large number of our archives had new permalinks that were now one character different from the original. In fact, all of our most popular permalinks were affected. Bookmarks, inbound links and search results were all getting a 404 page, and not a very good one at that.

Much of this traffic is lost forever. After correcting most of the permalinks traffic is up to approximately 20% of our pre-migration levels. I expect more of it to return as search engine crawlers clean up after the mess I caused. But it won’t all return and it will take months to rebuild to pre-migration levels. If our business depended on web site traffic, I would likely be updating my resumé now instead of writing this.

Lessons Learned

I learned these lessons the hard way so you don’t have to.

• Compare the permalinks from the old and the new CMS. All of them.
• Look at your web stats reports regularly.
• Make sure you have a helpful 404 page. One that says “404: Page Not Found” and nothing else is not helpful.
• Tag and categorize your content. The 404 page for the WordPress theme I’m using uses this information to help a visitor landing on the 404 page navigate to somewhere more useful.
• Get a real-time report of 404 errors. There are several WordPress plugins that will help. I used the 404 Notifier plugin by Alex King. That plugin sends an email anytime a 404 error occurs, but there are other plugins available that serve a similar purpose. If I had this kind of real-time report during the migration, the problem would have been identified and corrected much sooner and the traffic impact would have been minimized.
• Use a plugin that tries to locate the missing content. I’m using Smart 404 for WordPress by Michael Tyson. Smart 404 tries to find the missing page and redirects to the correct page. If it can’t locate the correct page then it tries send the visitor to a relevant category or tag. There are similar plugins that take other approaches, such as using a site map and a local Google search to locate either the missing content or relevant content.

The solution I found was faad and faac – Freeware Advanced Audio Decoder/Coder. It will decode unencrypted AAC/MP4 files and will happily ignore and correct the munged frame duration in the MP4 wrapper. I recovered a damaged track like this:

faad "09 This Devil's Workday.m4a" -o "09 This Devil's Workday.wav"

mv "09 This Devil's Workday.m4a" "09 This Devil's Workday.m4a.bad"

faac -w --artist "Modest Mouse" --title "This Devil's Workday" --album "Good News for People Who Love Bad News" --track 9 --disc 1 --year 2004 "09 This Devil's Workday.wav"

As faad was decoding the original track it showed that the MP4 wrapper’s duration was, but it corrected as it decoded it.

MP4 seems to have incorrect frame duration, using values from AAC data.
Decoding 09 This Devil's Workday.m4a took: 1.62 sec. 85.94x real-time.

When faad/faac were done I was able to re-import the track into iTunes and play it successfully.

Caveats

I lost some of the iTunes metadata from the original track, mainly the cover art. iTunes automatically added the cover art when I re-imported it, but that only works for music in the iTunes library. In those cases, faac has an option to import cover art from an external image file. You can do the same thing with iTunes as well.

If you have a lot of tracks to repair, adding the metadata manually on the command line is tedious. However, it could be automated with a simple script that captures and processes the output of faad since it displays all the track metadata as it is processing.

I put a band-aid over this problem by adding /dev/pty to /etc/fstab. After revisiting this bug report I found that the source of the problem was that I was missing a symlink in /etc/rcS.d. If you are having the same problem first check that
/etc/init.d/mountdevsubfs.sh exists and then run the following commands:

cd /etc/rcS.d/

sudo ln -s ../init.d/mountdevsubfs.sh S11mountdevsubfs.sh

However, there is no explanation for why this symlink was missing in the first place.

No RAID

This turned out to be an extremely easy problem to fix. While fixing the symlink problem above I noticed that init scripts weren't ordered correctly. The init scripts for disk mounting were happening too early, before the mdadm module and services had fully loaded. That's why my RAID volume wasn't mounting on startup but I could mount it manually after logging in. I corrected the order of the init scripts and now the RAID volume auto-mounts when booting.