Technical Exhaustion

I’ve always found CPAN modules troubling.

CPAN is a great, easy way to quickly get a Perl module onto the system. But unless you’re lucky and that module is available elsewhere using the native systems package manager (i.e rpm, ports or similar) then the CPAN installs become divorced from the system perl as time goes on. Ending up in conflicts and pain. It’s not the end of the world to keep these two in step but sometimes simpler is better and it’s nicer, as a user, to just drop the perl modules from CPAN you need into your own home directory (or other, non system, work area).

Of course, sometimes you just don’t have root! And need that module right now!

It turns out this is easy to do with a little digging through the Perl and CPAN docs.

• Set the CPAN makepl_arg flag to take a PREFIX= argument.
• Install your modules
• Point Perl to include your new module location in it’s search path

Setting the makepl_arg flag can be done during the first run of CPAN when it asks you a whole bunch of questions most of which, at the time, don’t mean much to you. The alternative way to do it (which is reproducible) is to run CPAN and configure it from within.

cpan[1]> o conf makepl_arg PREFIX=~/perl_modules
makepl_arg         [PREFIX=/usr/home/<user>/perl_modules]

Then commit the change to make CPAN write out your updated config.

cpan[2]> o conf commit
commit: wrote ‘/usr/home/<user>/.cpan/CPAN/MyConfig.pm’

Now we can just install CPAN modules as normal. Now they’ll be built and installed into the directory we’ve configured above.

The last thing we have to do is to tell perl that, whenever we run it, it should look in this new directory to find modules as well as in the global system module include paths.

We can do this by manipulating the PERL5LIB environment variable.

export PERL5LIB=/usr/home/<user>/perl_modules/lib/perl5/site_perl/5.8.8

The example above is in bash – adjust according to your favorite shell! It is probably also worth setting this in your environment at login using whatever method your shell uses.

That’s it! You can verify things look better by using the two commands :

perl -V

Which will display the @INC path perl has for finding modules. Your new local module directory should show up there.

You can also query CPAN to make sure that it’s happy that the modules you’ve install using it are actually installed.

perl -MCPAN -e autobundle

This just shows it’s own library but is useful for confirmation.

I’ve previously discussed how to recover from a disk failure in gmirror. It’s probably a good idea to describe how to set up a new mirror from scratch.

Rather handily this can be done on a live system, remotely via ssh in the space of a few minutes. One reboot is required but other than that the process is seamless.

My existing install is on disk ad4 and the ‘spare’ disk I want to mirror this existing installation to is ad6.

Both are 160 gig SATA disks.

Login in as root however way you see fit.

Before we can configure the new mirror we need to disable a safety feature of GEOM :

# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16

This allows us to perform operations on a mounted, and in use disk. This is just set for the current session and will be reset to normal when we reboot – which we’ll be doing shortly.

Whilst we’re doing housekeeping, lets enable gmirror at boot :

# echo geom_mirror_load="YES" >> /boot/loader.conf

You should probably check your loader.conf first. On a fresh system its empty and the above command is safe, however you probably want to have a quick glance over it and backup your existing one as a precaution if it contains existing settings.

Lets create our new mirror – with one command :

# gmirror label -v -b round-robin gm0 /dev/ad4
Metadata value stored on /dev/ad4

And we’re done! We now have a gmirror device but with only one member – the existing disk.

There are a number of different balance algorithms to choose from, the gmirror man page steps in :

load, prefer,round-robin,split

Finding out information on how these actually perform under the hood (beyond the superficial explanation in the man page) is quite tricky. And indeed most information leads to the fact that they don’t really behave as hoped and you’re better off sticking with round-robin which does what it says and gives marginally the best performance.

If you want to test with your specific work load / pattern you can change this anytime you like :

gmirror configure -b load gm0

Now we have a mirror device, and have enabled the drivers to be loaded on boot – we need to tell our system to mount from it next boot. So we need to edit /etc/fstab. Take a backup of your existing one now!

What happens next depends on your specific install. In general you need to add /dev/mirror/ to each entry and you need to change your hard disk device to be gm0 (or whatever you chose in the above to call your mirror device).

So I had a previous entry :

/dev/ad4s1a / ufs rw 1 1

Which now becomes :

/dev/mirror/gm0s1a / ufs rw 1 1

Repeat for each mount point that contains reference to your existing disk. Make sure you get this right, any errors here means you won’t be booting back up without difficulty!

Once you’re happy – reboot!

Cross your fingers.

Wait anxiously.

Nervously fidget.

And you’re back! Log in and do a quick df, hopefully you should see something akin to :

/dev/mirror/gm0s1a 507630 141610 325410 30% /

Looking good so far. Lets check the mirror is up :

# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad4

ok, so far so good. We’re up and running on our new mirror device. But it still only has one member. We need to add our second, ad6, disk.

#gmirror insert gm0 /dev/ad6

And we’re done! You’re now mirrored – your mirror is currently degraded :

# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad4
ad6 (70%)

But will soon become complete when the new disk finished sync’ing.

# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad4
ad6

And we’re done.

There are some caveats that should be mentioned.

I’ve run this config in anger in production for years now and never had problems – but there is a more complicated procedure to configure gmirror :

http://people.freebsd.org/~rse/mirror/

This takes into account the differening gemoetries of your two disks and avoids any problem with data going where it shouldn’t.

http://markmail.org/message/6vqfrgufv6jeu3yq#query:freebsd%20set%20up%20geom%20mirror+page:1+mid:wfkmluqgni45jzdo+state:results

gmirror also seems to be a performance dog, especially when compared to Linux software RAID. Which is unfortunate.

http://www.freebsdwiki.net/index.php/RAID,_performance_tests

But it’s ease of use lets me put up with that. Although hopefully we will see improvements over the next few versions of FreeBSD.

I came across a known bug in the FreeBSD installer for the first time today.

During the text network install (using the netboot iso) I had problems resolving the hostname of my chosen FTP mirror. The installer hung for an age trying to connect.

Assuming I had a made a typo in the network config, I ctrl+c’d the installer and chose to restart the install process when prompted.

This time round I connected and started retrieving package sets correctly – maybe I did make that typo after all.

However in the half dozen or so times I rebooted the server and restarted the install completely from scratch – I received various different errors ranging from simple ‘disk full’ messages to ‘unable to create symlink’ errors and even ‘no free inodes’ !

I was working remotely over a KVMoIP at the time and was starting to fret I had hardware issues on the other end of the line.

This belief was also founded on receiving the notorious disk geometry error from the installer during the partitioning stage. This is a long standing ‘issue’ and these days really means very little (in fact I believe it is being supressed as of FreeBSD 8? Finally!) and from past experience usually presents no problem – but there is always a nagging doubt – especially if more things go wrong!

However it turns out I had fallen foul of yet another long standing bug in the installer. This time if ctrl+c’ing out the installer and restarting (without rebooting) the next time round the installation attempts to write to the ramdisk instead of your allocated spinning disk – which obviously fills up very quickly and soon gives a variety of disk full errors!

http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/45565
http://www.freebsd.org/cgi/query-pr.cgi?pr=42162&cat=bin

These reports are dated 2002! and still this issue remains.

The solution is to point the installer to the IP address of the FTP server and bypass the icky host resolution.

FreeBSD has a ‘notorious’ reputation for being very hardware picky which, in actual fact, is quite far from the truth these days. Whilst not matching Linux, support is very good – certainly when compared to OpenSolaris. However I can’t help but wonder if these two very long standing features / bugs which present themselves very quickly during the already outdated (by modern standards) install procedure, cause people to instantly assume there is a support issue with their disk and / or controller and thus to move on (quite speedily I would warrant) from FreeBSD.

There is constant talk of a revamped installer but as of yet no sign. Let’s hope FreeBSD 8 brings some improvement.

Some quick Solaris 10 Zone management commands.

To list your running zones :

# zoneadm list

And with a bit more detail :

#zoneadm list -v

Change the IP address of a Zone

#zonecfg -z
zonecfg:zonename> select net address=oldip
zonecfg:zonename:net> set address=newip
zonecfg:zonename:net> end
zonecfg:zonename> commit

Change the name of a Zone

#zonecfg -z zonename
zonecfg:zonename> set zonename=newzonename
zonecfg:zonename> commit
zonecfg:zonename> exit

Moving a Zone on disk

Stop the zone (let do this cleanly) :

#zoneadm -z zonename halt

Then move :

#zoneadm -z zonename move /your/new/path

You should see a message similar to :

Moving across file systems; copying zonepath /your/new/path...
Cleaning up zonepath /your/new/path...

And bring it back up :

#zoneadm -z zonename boot

First a caveat : the following steps worked well on a lightly used mySQL server in production (~5 queries per second average, heavily biased to select’s – spread over around 30 individual databases) but your experience may vary on more heavily loaded servers. This also uses the ports build and the portupgrade tool. mySQL 5 also brings changes to table structures, new features and changed features within the server. Ensure your applications or other database users are compatible with 5 before proceeding. Test accordingly!

Canonical steps :

- Use mysqldump or whatever other backup process you have in place to take a backup before you begin. Once the backup has been taken move it somewhere safe and mark it as being taken immediately before the upgrade from 4 to 5. The upgrade process from 4 to 5 will examine/alter the structure and elements of *all* your databases so this is a significant backup milestone to keep.

- We have to upgrade the client libraries first. As the mySQL 4 and mySQL 5 ports are completely seperate in the ports tree, we will use portupgrade to install 5, replacing 4 along the way. This is a different process than simply updating a port as you would normally.

cd /usr/ports
portupgrade -o databases/mysql51-client mysql-client

This should install the upgraded console and libraries. You can verify the version with :

# mysql --version
mysql Ver 14.14 Distrib 5.1.30, for portbld-freebsd6.2 (i386) using 5.0

A good idea at this point is to check your server is still happy – run some queries and poke it a little bit.

- The big one! We essentially do the same steps as the client upgrade. This will upgrade whilst the server continues to run. However, for some reason, the port install script stops mysql during the process – so this is the start of your, hopefully short, period of downtime.
cd /usr/ports
portupgrade -o databases/mysql51-server mysql-server

- If all has gone well you can now start your new mySQL 5 server.
# /usr/local/etc/rc.d/mysql-server start
The rc.conf options have not changed so this will start right up. Login and confirm the new servers version. Handily this also checks that you *can* still login – which is an excellent first step to pass!

# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2966
Server version: 5.1.30 FreeBSD port: mysql-server-5.1.30

- This updates the mySQL table structure and various other housekeeping tasks to migrate your data from mySQL 4 to 5 format. The official definition is thus :

mysql_upgrade program that checks all tables for incompatibilities with the current version of MySQL Server and repairs them if necessary.

Strangely we do need to have the server running (with the old out of date formatted data) to perform this update – but mySQL seems quite happy to do so. Be warned this does change the data in your main mySQL database and it also parses and updates all your other databases where necessary. Hope you took that backup!
#mysql_upgrade -u root -p –datadir=/var/db/mysql
Obviously amend the datadir path if necessary – although the above is the FreeBSD default. This will grind away for potentially some time depending on the size of your data.

- That’s it, you’re all done! Now test, test and test. Check the data integrity and check your applications are still behaving as expected. You should also take the chance to update the client tools of other servers who may access this to version parity. It would also be worth rebuilding any ‘middleware’ ports you have that interface with mySQL (php5-mysql, p5-mysql etc).

Have fun and good luck!

To find a list of current SGE projects, use the following flags to qconf :

qconf -sprjl

Which can be conveniently remembered as ‘show project list’.

A nice cheat sheet for some common SGE tasks :

http://idolinux.blogspot.com/2008/09/sge-cheat-sheet.html

Quick and dirty – this should never be needed, but in the rare events that your auto rotation cycle fails or you’re just using a volume for testing and want to blank it to start from scratch, from the Bacula console enter :

purge jobs volume

After a short warning you can choose which pool and then which volume you want to purge.

Running software RAID-1 using gmirror under FreeBSD can save some headaches.

Sometimes a disk literally fails leaving you running on a spare, sometimes you just have a ‘blip’ during a busy period of the server which causes gmirror to drop the disk from the array.

This happened today and from dmesg we can see :

GEOM_MIRROR: Request failed (error=5). ad4[READ(offset=48103390720, length=16384 )]
GEOM_MIRROR: Device gm0: provider ad4 disconnected.

Basically, my SATA disk ad4 had a bit of a read error. As this has happened before I’m not overly concerned. I don’t know, however, if this is a physical disk problem or just gmirror err’ing on the side of caution and aggressively removing it from the array. A full s.m.a.r.t report in due course may shed some light. If this had been a write error I would be a bit more concerned. Nonetheless gmirror caught it and removed it from the array as faulty.

We can confirm using the gmirror list and gmirror status commands

guru# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad6

guru# gmirror list
Geom name: gm0
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 6
SyncID: 1
ID: 2879715010
Providers:
1. Name: mirror/gm0
Mediasize: 79999999488 (75G)
Sectorsize: 512
Mode: r5w5e6
Consumers:
1. Name: ad6
Mediasize: 80026361856 (75G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: DIRTY
GenID: 6
SyncID: 1
ID: 1663191490

Which tells us the mirror device gm0 is degraded and only SATA disk ad6 is still playing.

In an actual disk failure situation at this point we’d bring the server down and replace the failed ad4 device. If hardware supports you could also hotswap – but I’m not sure how the FreeBSD kernel would handle that.

We don’t need to do that in this instance, as I’m fairly sure this is just a blip. So I’ll just re add the disk to the array and see if it rebuilds.

After replacing the disk (if necessary) to re add the disk we first have to tell gmirror to ‘forget’ components of the array that have failed.

guru# gmirror forget gm0

This forgets the broken members for the array ‘gm0′.

We can confirm using the gmirror status command :

guru# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad6


Now we can add (or re-add!) the ‘new’ disk and watch it rebuild!

guru# gmirror insert gm0 ad4
guru# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad6
ad4 (0%)

After a while…

guru# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad6
ad4 (64%)

And finally…

guru# gmirror status
Name Status Components
mirror/gm0 COMPLETE ad6
ad4
guru#


We can glean a nice summary from dmesg :

GEOM_MIRROR: Device gm0: provider ad4 detected.
GEOM_MIRROR: Device gm0: rebuilding provider ad4.
GEOM_MIRROR: Device gm0: rebuilding provider ad4 finished.
GEOM_MIRROR: Device gm0: provider ad4 activated.

I unwittingly stumbled across a reasonably well hidden Bacula design foible today.

I’ve noticed for some time that when doing test restores, or browsing the ‘most recent backup’ for a client Bacula has included files that should not be there based on the timeline of the restore.

If, after performing a full backup, some files are deleted and then a differential (or incremental!) backup is taken these previously deleted files would still be restored if I did the ‘latest backup’ for a client.

The expected behavior (in my opinion) would be for these files not to be included in a restore of the most recent backup as, at the point in time the latest backup was taken, the files were not on disk.

This is a known feature of Bacula and is documented as a sub-project to fix

This has been underway since 2005, hopefully good progress is being made.

Unfortunately this means that any restores made expecting a ‘point in time’ snapshot of the system to be written back to disk will not behave as expected.

This may leave the system in an inconsistent or, at best, unknown state. This will also skew disk usage for a restore, using more space than expected and may either cause problems depending on the target disk size or necessitate a spring clean post restore to tidy things up again. This would be particularly painful for administrators with busy systems and lots of changes between incremental and differential backup windows.

This only occurs between each full backup so one workaround would be to perform a full backup each day but this is not often a practical solution.

It’s some comfort that at least the data is backed up safely and that data can be retrieved in some form, however this makes the restore process a little more complicated than it perhaps should be.