Technical Exhaustion

I’ve previously discussed how to recover from a disk failure in gmirror. It’s probably a good idea to describe how to set up a new mirror from scratch.

Rather handily this can be done on a live system, remotely via ssh in the space of a few minutes. One reboot is required but other than that the process is seamless.

My existing install is on disk ad4 and the ’spare’ disk I want to mirror this existing installation to is ad6.

Both are 160 gig SATA disks.

Login in as root however way you see fit.

Before we can configure the new mirror we need to disable a safety feature of GEOM :

# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16

This allows us to perform operations on a mounted, and in use disk. This is just set for the current session and will be reset to normal when we reboot - which we’ll be doing shortly.

Whilst we’re doing housekeeping, lets enable gmirror at boot :

# echo geom_mirror_load="YES" >> /boot/loader.conf

You should probably check your loader.conf first. On a fresh system its empty and the above command is safe, however you probably want to have a quick glance over it and backup your existing one as a precaution if it contains existing settings.

Lets create our new mirror - with one command :

# gmirror label -v -b round-robin gm0 /dev/ad4
Metadata value stored on /dev/ad4

And we’re done! We now have a gmirror device but with only one member - the existing disk.

There are a number of different balance algorithms to choose from, the gmirror man page steps in :

load, prefer,round-robin,split

Finding out information on how these actually perform under the hood (beyond the superficial explanation in the man page) is quite tricky. And indeed most information leads to the fact that they don’t really behave as hoped and you’re better off sticking with round-robin which does what it says and gives marginally the best performance.

If you want to test with your specific work load / pattern you can change this anytime you like :

gmirror configure -b load gm0

Now we have a mirror device, and have enabled the drivers to be loaded on boot - we need to tell our system to mount from it next boot. So we need to edit /etc/fstab. Take a backup of your existing one now!

What happens next depends on your specific install. In general you need to add /dev/mirror/ to each entry and you need to change your hard disk device to be gm0 (or whatever you chose in the above to call your mirror device).

So I had a previous entry :

/dev/ad4s1a / ufs rw 1 1

Which now becomes :

/dev/mirror/gm0s1a / ufs rw 1 1

Repeat for each mount point that contains reference to your existing disk. Make sure you get this right, any errors here means you won’t be booting back up without difficulty!

Once you’re happy - reboot!

Cross your fingers.

Wait anxiously.

Nervously fidget.

And you’re back! Log in and do a quick df, hopefully you should see something akin to :

/dev/mirror/gm0s1a 507630 141610 325410 30% /

Looking good so far. Lets check the mirror is up :

# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad4

ok, so far so good. We’re up and running on our new mirror device. But it still only has one member. We need to add our second, ad6, disk.

#gmirror insert gm0 /dev/ad6

And we’re done! You’re now mirrored - your mirror is currently degraded :

# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad4
ad6 (70%)

But will soon become complete when the new disk finished sync’ing.

# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad4
ad6

And we’re done.

There are some caveats that should be mentioned.

I’ve run this config in anger in production for years now and never had problems - but there is a more complicated procedure to configure gmirror :

http://people.freebsd.org/~rse/mirror/

This takes into account the differening gemoetries of your two disks and avoids any problem with data going where it shouldn’t.

http://markmail.org/message/6vqfrgufv6jeu3yq#query:freebsd%20set%20up%20geom%20mirror+page:1+mid:wfkmluqgni45jzdo+state:results

gmirror also seems to be a performance dog, especially when compared to Linux software RAID. Which is unfortunate.

http://www.freebsdwiki.net/index.php/RAID,_performance_tests

But it’s ease of use lets me put up with that. Although hopefully we will see improvements over the next few versions of FreeBSD.

I came across a known bug in the FreeBSD installer for the first time today.

During the text network install (using the netboot iso) I had problems resolving the hostname of my chosen FTP mirror. The installer hung for an age trying to connect.

Assuming I had a made a typo in the network config, I ctrl+c’d the installer and chose to restart the install process when prompted.

This time round I connected and started retrieving package sets correctly - maybe I did make that typo after all.

However in the half dozen or so times I rebooted the server and restarted the install completely from scratch - I received various different errors ranging from simple ‘disk full’ messages to ‘unable to create symlink’ errors and even ‘no free inodes’ !

I was working remotely over a KVMoIP at the time and was starting to fret I had hardware issues on the other end of the line.

This belief was also founded on receiving the notorious disk geometry error from the installer during the partitioning stage. This is a long standing ‘issue’ and these days really means very little (in fact I believe it is being supressed as of FreeBSD 8? Finally!) and from past experience usually presents no problem - but there is always a nagging doubt - especially if more things go wrong!

However it turns out I had fallen foul of yet another long standing bug in the installer. This time if ctrl+c’ing out the installer and restarting (without rebooting) the next time round the installation attempts to write to the ramdisk instead of your allocated spinning disk - which obviously fills up very quickly and soon gives a variety of disk full errors!

http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/45565
http://www.freebsd.org/cgi/query-pr.cgi?pr=42162&cat=bin

These reports are dated 2002! and still this issue remains.

The solution is to point the installer to the IP address of the FTP server and bypass the icky host resolution.

FreeBSD has a ‘notorious’ reputation for being very hardware picky which, in actual fact, is quite far from the truth these days. Whilst not matching Linux, support is very good - certainly when compared to OpenSolaris. However I can’t help but wonder if these two very long standing features / bugs which present themselves very quickly during the already outdated (by modern standards) install procedure, cause people to instantly assume there is a support issue with their disk and / or controller and thus to move on (quite speedily I would warrant) from FreeBSD.

There is constant talk of a revamped installer but as of yet no sign. Let’s hope FreeBSD 8 brings some improvement.

First a caveat : the following steps worked well on a lightly used mySQL server in production (~5 queries per second average, heavily biased to select’s - spread over around 30 individual databases) but your experience may vary on more heavily loaded servers. This also uses the ports build and the portupgrade tool. mySQL 5 also brings changes to table structures, new features and changed features within the server. Ensure your applications or other database users are compatible with 5 before proceeding. Test accordingly!

Canonical steps :

- Use mysqldump or whatever other backup process you have in place to take a backup before you begin. Once the backup has been taken move it somewhere safe and mark it as being taken immediately before the upgrade from 4 to 5. The upgrade process from 4 to 5 will examine/alter the structure and elements of *all* your databases so this is a significant backup milestone to keep.

- We have to upgrade the client libraries first. As the mySQL 4 and mySQL 5 ports are completely seperate in the ports tree, we will use portupgrade to install 5, replacing 4 along the way. This is a different process than simply updating a port as you would normally.

cd /usr/ports
portupgrade -o databases/mysql51-client mysql-client

This should install the upgraded console and libraries. You can verify the version with :

# mysql --version
mysql Ver 14.14 Distrib 5.1.30, for portbld-freebsd6.2 (i386) using 5.0

A good idea at this point is to check your server is still happy - run some queries and poke it a little bit.

- The big one! We essentially do the same steps as the client upgrade. This will upgrade whilst the server continues to run. However, for some reason, the port install script stops mysql during the process - so this is the start of your, hopefully short, period of downtime.
cd /usr/ports
portupgrade -o databases/mysql51-server mysql-server

- If all has gone well you can now start your new mySQL 5 server.
# /usr/local/etc/rc.d/mysql-server start
The rc.conf options have not changed so this will start right up. Login and confirm the new servers version. Handily this also checks that you *can* still login - which is an excellent first step to pass!

# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2966
Server version: 5.1.30 FreeBSD port: mysql-server-5.1.30

- This updates the mySQL table structure and various other housekeeping tasks to migrate your data from mySQL 4 to 5 format. The official definition is thus :

mysql_upgrade program that checks all tables for incompatibilities with the current version of MySQL Server and repairs them if necessary.

Strangely we do need to have the server running (with the old out of date formatted data) to perform this update - but mySQL seems quite happy to do so. Be warned this does change the data in your main mySQL database and it also parses and updates all your other databases where necessary. Hope you took that backup!
#mysql_upgrade -u root -p –datadir=/var/db/mysql
Obviously amend the datadir path if necessary - although the above is the FreeBSD default. This will grind away for potentially some time depending on the size of your data.

- That’s it, you’re all done! Now test, test and test. Check the data integrity and check your applications are still behaving as expected. You should also take the chance to update the client tools of other servers who may access this to version parity. It would also be worth rebuilding any ‘middleware’ ports you have that interface with mySQL (php5-mysql, p5-mysql etc).

Have fun and good luck!

Running software RAID-1 using gmirror under FreeBSD can save some headaches.

Sometimes a disk literally fails leaving you running on a spare, sometimes you just have a ‘blip’ during a busy period of the server which causes gmirror to drop the disk from the array.

This happened today and from dmesg we can see :

GEOM_MIRROR: Request failed (error=5). ad4[READ(offset=48103390720, length=16384 )]
GEOM_MIRROR: Device gm0: provider ad4 disconnected.

Basically, my SATA disk ad4 had a bit of a read error. As this has happened before I’m not overly concerned. I don’t know, however, if this is a physical disk problem or just gmirror err’ing on the side of caution and aggressively removing it from the array. A full s.m.a.r.t report in due course may shed some light. If this had been a write error I would be a bit more concerned. Nonetheless gmirror caught it and removed it from the array as faulty.

We can confirm using the gmirror list and gmirror status commands

guru# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad6

guru# gmirror list
Geom name: gm0
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 6
SyncID: 1
ID: 2879715010
Providers:
1. Name: mirror/gm0
Mediasize: 79999999488 (75G)
Sectorsize: 512
Mode: r5w5e6
Consumers:
1. Name: ad6
Mediasize: 80026361856 (75G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: DIRTY
GenID: 6
SyncID: 1
ID: 1663191490

Which tells us the mirror device gm0 is degraded and only SATA disk ad6 is still playing.

In an actual disk failure situation at this point we’d bring the server down and replace the failed ad4 device. If hardware supports you could also hotswap - but I’m not sure how the FreeBSD kernel would handle that.

We don’t need to do that in this instance, as I’m fairly sure this is just a blip. So I’ll just re add the disk to the array and see if it rebuilds.

After replacing the disk (if necessary) to re add the disk we first have to tell gmirror to ‘forget’ components of the array that have failed.

guru# gmirror forget gm0

This forgets the broken members for the array ‘gm0′.

We can confirm using the gmirror status command :

guru# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad6


Now we can add (or re-add!) the ‘new’ disk and watch it rebuild!

guru# gmirror insert gm0 ad4
guru# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad6
ad4 (0%)

After a while…

guru# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad6
ad4 (64%)

And finally…

guru# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad6
ad4
guru#


We can glean a nice summary from dmesg :

GEOM_MIRROR: Device gm0: provider ad4 detected.
GEOM_MIRROR: Device gm0: rebuilding provider ad4.
GEOM_MIRROR: Device gm0: rebuilding provider ad4 finished.
GEOM_MIRROR: Device gm0: provider ad4 activated.