Using Mondorescue to P2V old versions of Linux - Part 1 Backup and Recovery to same host

We have recently had a situation where a client had a number of very very old linux boxes. Some of these servers were pushing 10 years old

A lot of these systems are only online for historical or political reasons so the quickest (not necessarily the best) way to reduce the hardware risk was to P2V them. As we were already using Mondo to back these up its fairly straightforward to use Mondo as a P2v Tool.

NB These are fairly rough and ready articles and there will be one for each version on linux we used.

They all assume that you know what you are doing. This first article covers the configuration of Mondo allowing you to backup and recover to the same hardware.

We used mondo to backup to a central NFS mount on a Netapp filer using the following approach;

The backups are performed via two scripts;

/usr/local/bin/mondomount.sh

This script mounts the NFS share on the SAN to /mnt/nfs. If directories are missing they are created. This script is local to each host to cater for different network conditions. Typically the address used is SAN01Node1vif0.domain.local (192.168.XX.XX) however for hosts at the external Co Lo site there is a NAT translation on the router to aaa.bbb.ccc.dd. This is masked in the script by using the hostname of SAN01Node1vif0.domain.local

For servers outside the router (ie public IP addresses) there is a local /etc/hosts entry that maps SAN01Node1vif0.domain.local to aaa.bbb.ccc.dd

This script executes a minute before the main backup script.

The main backup script is;

/mnt/nfs/scripts/mondobackup.sh

This script takes a full backup of each host excluding NFS mounts and places them on /mnt/nfs/$HOSTNAME/mondo/$DATE.

A new directory is created for each day. The script also copies the mondoarchive.log file and mondorescue.iso to the same location.
It will also delete old backups (currently 1 day is retained) and email a success or fail message to logs@domain.local

As all servers run a single script it is important that changes to this script are fully tested.

The script is execute via CRON using the following entries
#Mondo backup to SAN
5 20 * * 1-5 /usr/local/bin/mondomount.sh
7 20 * * 1-5 /mnt/nfs/scripts/mondobackup.sh
Currently the script commences at 20:05 Monday to Friday

ADDITION OF NEW HOSTS

To migrate a host to this configuration use the steps below;

Install mondo and any prerequisites, this script requires mondo 2.2.4 or greater

mount -t nfs 192.168.XX.XX:/vol/VOLNAME /mnt/
cp /mnt/openfiler/scripts/mondomount.sh /usr/local/bin
mondomount.sh
crontab -e

Add the following lines

#Mondo backup to SAN
5 20 * * 1-5 /usr/local/bin/mondomount.sh
7 20 * * 1-5 /mnt/nfs/scripts/mondobackup.sh

NB the mount directory in the first line can be any valid directory for mounting a directory.

Known Issues

The script contains different mondo command entries depending on which version of Fedora Core is being used, this was originally due to multiple versions of mondo being installed and the quirks of each version. The bulk of these have been resolved however mondo version 2.2.4 and 2.2.9 require slightly different command lines for the NFS mount directory. The code to handle multiple versions has been left in to provide future flexibility if required.

Failing to use the appropriate NFS command options (for example simply creating the ISO’s on a specific directory instead of the nfs command option) may result in valid data that cannot be easily restored. Using the nfs options ensures all the data paths to the backup files are embedded in the mondorescue recovery CD

Recovery Procedure
Mondo utilises a mondorescue CD which is a standard ISO image created as part of each backup and copied to the NFS directory. This ISO needs to be downloaded using winscp or similar and burnt to a CD/DVD. From here the recovery can be commenced.

Once the CD is inserted into the machine allow it to boot. At the boot: prompt enter interactive.

The CD will then start up a cutdown version of linux , start the network stack and the like. It is important to note that the CD will start the host up on the same IP address as it normally has. This is not an issue if you are in a DR scenario however if you are testing a recovery on equivalent hardware, or performing a P2V you may end up with duplicate IP’s on the network. If this is the case change the command line on the boot prompt to

Interactive ipconf=eth0:dhcp

Once the operating system has loaded mondorestore should start automatically.

Net mount
Leave the network protocol as default
Leave the network share as default
Leave the directory as default
Leave the prefix as default
Leave the mountlist as default and click OK
Select yes on the warning the mountlist might not work
Select no on erase and partition hard drives
Select yes to formatting hard drives
Select yes to restore all data
Allow the restore to complete
You are then prompted to edit multipath.conf
Save the file
You are then prompted to initiliaze the boot loader, select yes
Select n to initialise initrd, this is not required
Select no to modify/clone system
Type reboot to restart the server

If you are changing anything such as hard disk layouts it is recommended you boot in expert mode and manually partition the disks. Once the restore is completed you can use the mount-me script to mount the recovered partitions under /mnt/RESTORING and make any other filesystem changes required.

The server should restart cleanly however if there any issues you may need to reboot the mondo rescue CD using the boot: string of

expert

and then resolve these issues.

Mondomount.sh script
Below is the script that mounts the NFS directory that is run just prior to the mondobackup. It is installed locally to each machine.

Please note we modified this script as a subset of the mondobackup.sh script which someone kindly posted on the web and we modified to suit our own purposes. There is still scope to clean these up however they do work OK

#!/bin/bash
#mounts the NFS share to allow centralised script to run

## declare variables
HOSTNAME=`hostname`
#NFS_HOST="192.168.1.12"
NFS_HOST="SAN01Node1vif0.domain.local"
NFS_DIRECTORY="/vol/VOLUME"
#NFS_HOST should be in the format 192.168.1.12 for V3
DATE=`date +%F`
DEST="/mnt/nfs"
#DEST should be in the format /mnt/nfs
#DEST is the local directory mounted to the NFS host

#Confirm the DEST exists
echo -n "Checking that the NFS mount directory exists: "

if [[ -d $DEST ]]; then
echo NFS Mount directory exists!
else
mkdir $DEST
if [[ $? != 0 ]]; then
echo Failed! Something is wrong.
exit 1
else
echo "Mount Directory did not exist - created it."
fi
fi

#Mounting NFS mount
umount $DEST > /dev/null 2>&1
mount -t nfs $NFS_HOST:$NFS_DIRECTORY $DEST

Monobackup.sh
This script executes the backup

#! /bin/bash
# updated 270611 KH for NFS on SAN and general clean up
#
# Mondo/Mindi unattended backup script
#
# User defined variables
## declare variables
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
export PATH
HOSTNAME=`hostname`
NFS_HOST="SAN01Node1vif0.domain.local"
#NFS_HOST should be in the format 192.168.1.12 for V3
NFS_DIRECTORY="/vol/nfs01sata"
#NFS_DIRECTORY is the mounted directory on the NFS host
#Should be in the format of "/vol/mountdir" or similar
DATE=`date +%F`
DEST="/mnt/nfs"
#DEST should be in the format /mnt/nfs
#DEST is the local directory mounted to the NFS host
#the NFS host is mounted via the mondomount.sh script
ISO_PREFIXNAME=mondo_"$HOSTNAME"_"$DATE"
MAX_OLBBACKUPS='1'
#MAX_OLDBACKUPS means that todays plus X days backups will be retained
#ie if the value is 1 then 2 days backups are retained, yesterdays and this one
BACKUP_DIR=$DEST/$HOSTNAME/mondo
# BACKUP_DIR = directory backups are written to(must include preceding "/"
# NB when backing up via NFS the $DEST must be removed from the mondoarchive line
# so the -d option is explicitly set to $HOSTNAME/mondo
# if you want a different directory both need to be modified
ADMIN=logs@domain.local # Admin email to send reports to
LOG=/var/log # Log file directory
EXCLUDE_MNT="/dev/shm" # List of *mount points* (not directories) to exclude from the backup process (seperated by white-space)
EXCLUDE_DIR="/windows" # List of *directories* (not mount points) to exclude from the backup process (seperated by white-space)
ISO_SIZE=4480m # Set the max size of the ISO files to be created
COMP=0 # Set the compression level (0-9; 0 = lowest, 9 = highest)
#SCRATCH_DIR=/tmp # Directory where ISOs will be created before archive

#Confirm the DEST exists
echo -n "Checking that the NFS mount directory exists: "

if [[ -d $DEST ]]; then
echo NFS Mount directory exists!
else
mkdir $DEST
if [[ $? != 0 ]]; then
echo Failed! Something is wrong.
exit 1
else
echo "Mount Directory did not exist - created it."
fi
fi

#Mounting NFS mount
#umount $DEST > /dev/null 2>&1
#mount -t nfs $NFS_HOST:$NFS_DIRECTORY $DEST

#Confirm the BACKUP_DIR exists
echo -n "Checking that the BACKUP_DIR directory exists: "

if [[ -d $BACKUP_DIR ]]; then
echo BACKUP_DIR directory exists!
else
mkdir -p $BACKUP_DIR
if [[ $? != 0 ]]; then
echo Failed! Something is wrong.
exit 1
else
echo "Backup Directory did not exist - created it."
fi
fi

#Delete old backups based on MAX_OLDBACKUPS
find $BACKUP_DIR/ -type d -iname "*-*-*" -mtime +$MAX_OLBBACKUPS -exec /bin/rm -rf {} \;

# ==================================================================================================
# SCRIPT CONTENTS - DO NOT EDIT BELOW THIS POINT
# ==================================================================================================

LOG_DATE=`date "+%b %e %R:%S"` # Formatted date command for log files
DIR_DATE=`date "+backup_%F"` # Formatted date command for creating backup directories

USER_DIR=$(cd $BACKUP_DIR; ls -ld | awk '{print $3}')
GROUP_DIR=$(cd $BACKUP_DIR; ls -ld | awk '{print $4}')

# Define locations of vital programs/utilities

MONDOARCHIVE=`which mondoarchive`
MINDI=`which mindi`
CHOWN=`which chown`
MKDIR=`which mkdir`

# Translate $DEST to something usable by awk

DEST_TRANS=$(echo $DEST | sed 's!/!\\/!g')

# This must be run as root since access to every file is required

echo -n "Checking that you are root: "

if [[ $UID != 0 ]]; then
echo You must be root!
exit 1
else
echo OK, you are root!
fi

# Check that a log file is available, if not create one in the
# directory specified in the user defined variables.

echo -n "Checking that log file exists: "

if [[ -e $LOG\/backup.log ]]; then
echo Log file exists!
else
touch $LOG/backup.log
if [[ $? != 0 ]]; then
echo Failed! Something is wrong.
exit 1
else
echo Log file did not exist - created it.
fi
fi

# Log that backup script has started

echo "$LOG_DATE BACKUP[$$]: BEGIN - Mondo automation script started." > $LOG/backup.log

# Check that $DEST is actually a mount point and not a directory
#if [[ ! $(df | awk '/'$DEST_TRANS'$/') ]]; then
if [[ ! $(df -a | awk '/'$DEST_TRANS'$/') ]]; then
echo "You must use a mount point, not a directory in your configuration for 'DEST'! Quitting..."
echo -e "Backup failed - incorrect configuration:\n\nYou must use a mount point, not a directory in your configuration for the DEST variable!" | mail -s "Backup failed - incorrect configuration" $ADMIN
echo "$LOG_DATE BACKUP[$$]: ERROR - Incorrect configuration - DEST variable must be a mount point" >> $LOG/backup.log
exit 1
fi

# Check to see if directory exists for storing backups
echo -n "Checking for backup root directory: "
if [[ -d $BACKUP_DIR ]]; then
echo Directory exists!
BACKUP_DIR=$BACKUP_DIR
else
echo -n "Directory doesn't exist, creating... "
$MKDIR $BACKUP_DIR
$CHOWN $USER_DIR.$GROUP_DIR $BACKUP_DIR/mondo
if [[ $? != 0 ]]; then
echo Failed!
echo -e "Backup failed - couldn't create backup root directory:\n\n$BACKUP_DIR/mondo." | mail -s "Backup failure - couldn't create directory" $ADMIN
echo "$LOG_DATE BACKUP[$$]: ERROR - Couldn't create backup root directory: $BACKUP_DIR" >> $LOG/backup.log
exit 1
else
echo Done!
echo Backup root directory: $BACKUP_DIR
BACKUP_DIR=$BACKUP_DIR
fi
fi

# Create new backup directory for current ISO images

echo -n "Creating directory to store today's backup: "
if [[ -d $BACKUP_DIR/$DIR_DATE ]]; then
echo Directory exists! Continuing...
else
$MKDIR $BACKUP_DIR/$DIR_DATE
$CHOWN $USER_DIR.$GROUP_DIR $BACKUP_DIR/$DIR_DATE
if [[ $? != 0 ]]; then
echo Failed!
echo -e "Backup failed - couldn't create directory:\n\n$BACKUP_DIR/$DIR_DATE." | mail -s "Backup failure - couldn't create directory" $ADMIN
echo "$LOG_DATE BACKUP[$$]: ERROR - Couldn't create backup directory: $BACKUP_DIR/$DIR_DATE." >> $LOG/backup.log
exit 1
else
echo Done!
echo Backup directory: $BACKUP_DIR/$DIR_DATE
fi
fi

# Actually start the Mondo backup

echo "$LOG_DATE BACKUP[$$]: BEGIN - Mondo backup process started." >> $LOG/backup.log

# Start mondoarchive job
#
# -O = Archive (not verify)
# -i = Backup to ISO images only (no removable medium)
# -E = Exclude directories from backup
# -N = Exclude all NFS mounts (only backup local drives)
# -d = Backup to this directory
# -s = Set maximum ISO size
# -(0-9) = Set compression level
# -F = Do not offer to write boot+data floppy disk images - images still created & stored
# -S = Scratch directory for building ISO images
#

#if grep -q "Core release 1" /etc/redhat-release; then
# echo "Backing up FC 1"
# $MONDOARCHIVE -OV -i -d $BACKUP_DIR/$DIR_DATE -N -$COMP -s $ISO_SIZE
#else
# echo "Not backing up FC 1"
# $MONDOARCHIVE -OV -i -d $BACKUP_DIR/$DIR_DATE -N -$COMP -s $ISO_SIZE -p$ISO_PREFIXNAME
#fi
#

echo "NFS host is $NFS_HOST"
echo "NFS Directory is $NFS_DIRECTORY"
echo "Backup DIR is $BACKUP_DIR"
echo "Directory Date is $DIR_DATE"
if grep -q "Core release 1" /etc/redhat-release; then
echo "Backing up FC 1"
#$MONDOARCHIVE -OV -i -d $BACKUP_DIR/$DIR_DATE -N -$COMP -s $ISO_SIZE -F
$MONDOARCHIVE -OV -n $NFS_HOST:$NFS_DIRECTORY -d $HOSTNAME/mondo/$DIR_DATE -N -$COMP -s $ISO_SIZE -F
elif grep -q "Core release 6" /etc/redhat-release; then
echo "Backing up FC 6"
$MONDOARCHIVE -OV -n nfs://$NFS_HOST:$NFS_DIRECTORY -d $HOSTNAME/mondo/$DIR_DATE -N -$COMP -s $ISO_SIZE -p$ISO_PREFIXNAME
elif grep -q "Core release 3" /etc/redhat-release; then
echo "Backing up FC 3"
#$MONDOARCHIVE -OV -n $NFS_HOST:$NFS_DIRECTORY -d $HOSTNAME/mondo/$DIR_DATE -N -$COMP -s $ISO_SIZE
$MONDOARCHIVE -OV -n nfs://$NFS_HOST:$NFS_DIRECTORY -d $HOSTNAME/mondo/$DIR_DATE -N -$COMP -s $ISO_SIZE -p$ISO_PREFIXNAME
else
echo "Not backing up FC 1 or FC 6"
#$MONDOARCHIVE -OV -n nfs://$NFS_HOST:$NFS_DIRECTORY -d $BACKUP_DIR/$DIR_DATE -N -$COMP -s $ISO_SIZE -p$ISO_PREFIXNAME
$MONDOARCHIVE -OV -n nfs://$NFS_HOST:$NFS_DIRECTORY -d $HOSTNAME/mondo/$DIR_DATE -N -$COMP -s $ISO_SIZE -p$ISO_PREFIXNAME
fi

if [[ $? != 0 ]]; then
echo There was a problem on $HOSTNAME... FAIL!
echo "$LOG_DATE BACKUP[$$]: ERROR - $HOSTNAME Mondo backup process FAIL ! " >> $LOG/backup.log
echo -e "Backup failed - Mondo backup process failed:\n\nCheck the log file:\n\n /var/log/mondo-archive.log\n\nBackup incomplete." >> $LOG/backup.log
mail -s "$HOSTNAME Mondo backup process FAIL " $ADMIN < $LOG/backup.log
exit 1
else
echo $HOSTNAME Mondo backup process complete !
echo ISOs located at: $BACKUP_DIR/$DIR_DATE
echo -e "$HOSTNAME Mondo backup process successful!\n\nBackup ISOs located at: $BACKUP_DIR/$DIR_DATE \n\nLog file: /var/log/mondo-archive.log" >> $LOG/backup.log
#mail -s "$HOSTNAME Mondo backup process OK " $ADMIN <$LOG/backup.log
echo "$LOG_DATE BACKUP[$$]: $HOSTNAME SUCCESS - Mondo backup process completed!" >> $LOG/backup.log
mail -s "$HOSTNAME Mondo backup process OK " $ADMIN <$LOG/backup.log
fi

#copy log file and mondorescue iso to same location as backup
#cp /var/log/mondoarchive.log $BACKUP_DIR/$DIR_DATE/
#cp /var/cache/mindi/mondorescue.iso $BACKUP_DIR/$DIR_DATE/

if grep -q "Core release 1" /etc/redhat-release; then
echo "Working on FC 1"
cp /var/log/mondoarchive.log $BACKUP_DIR/$DIR_DATE/
elif grep -q "Core release 6" /etc/redhat-release; then
echo "Working on FC 6"
cp /var/log/mondoarchive.log $BACKUP_DIR/$DIR_DATE/
elif grep -q "Core release 3" /etc/redhat-release; then
echo "Working on FC 3"
cp /var/log/mondoarchive.log $BACKUP_DIR/$DIR_DATE/
else
echo "Not working on FC 1 or FC 6"
cp /var/log/mondoarchive.log $BACKUP_DIR/$DIR_DATE/
fi

#copy mondorescue iso - NB older version put in different directory so localhost has symlink to match the line below
cp /var/cache/mindi/mondorescue.iso $BACKUP_DIR/$DIR_DATE/

So basically using the above two scripts allows you to backup and restore via NFS to the same host. In the next part(s) we will cover the P2V process.