VI3, Disaster Recovery and Business Continuity

If there's one big advantage of using VI3, it certainly is its ability to ensure business continuity. Previous versions have done a great job, but VI3 and its new features like Vmware Consolidated Backup (VCB), High Availability (HA) and Dynamic Resource Scheduler (DRS) are really pushing it one step further.

The problem is although technology is making it easier every day, mentality is still an issue. The company I'm currently working for is just another example of it.

How it should be

We all know DR/BCP projects should be oriented to business needs and expectations, that have to be identified, studied, agreed and documented. Platform's RPOs and RTOs have to be investigated. Costs for outages have to be calculated. Risks have to be identified and mitigated. Reliability for the whole project has to be tested and assessed.

How it is sometimes

Now my current project is going nowhere near that way. For budget, time and political reasons my current DR project is being managed exactly the other way around: "Here is the available infrastructure and applications. Come up with the best DR/BC solution you can."

At the DR site (at least I was given one) I'm currently working at the infrastructure level, and according to my project plan I'll be finishing by the end of this month. With what I was provided (so far) my intentions are to take advantage out of the Vmware 2.5.x already implemented infrastructure, recover the critical platforms to this site and assume a "business as usual" status 12 hours after disaster situation is declared.

Scenario

I should now provide a high level picture of what I have. My main production site is some 300 kms away which I'm connected to via a 10Mbit circuit.

I also lucky enough to have:

- an IBM xSeries (with 4 x Xeon 2.5GHz, 8Gb RAM and a 34Gb RAID5 volume)
- an already working SAN (old IBM FastT700 - renamed to DS4000 - and 2 x Brocade/IBM 16 port fiber switch)
- a bunch of Snap Appliances (models 4200 and 2200)

I started by rack-mounting everything followed by passing all network and fiber cables. Connected the console. Added 8 more 300Gb hard drives (making a total of 12) to the EXP enclosures and 4 x 34Gb old hard drives that I have left.

Storage

After wiping all the existing information on the SAN and disks, used IBM Storage Manager 9 to reconfigure all adapters, WWNs, groups, hosts and ports. Also configured 3 brand new RAID 5 arrays.

Each array holds one LUN ans was configured as follows:

Array 1 - made with the old 34Gb disks, has one hot spare disk, and it will hold less critical data like images and ISOs (total: 101,5Gb)

Array 2 - made with the 8 new 300Gb disks, shares one hot spare disk with array 3 and it will hold backups of critical VMs, ready to be started up (total: 1,9Tb)

Array 3 - made with the last 300Gb disks, will also hold VMs (total: 1.1Tb)

VirtualCentre 2 and License Server

I then provisioned an HP ML series server (with 2 x P4 2.7GHz, 2Gb of RAM and a total of 101Gb). This was installed with the usual HP tools and MS Windows 2003 Standard Edition. All updates and fixes were applied. Finally installed VirtualCentre 2 and License Server.

Note: The License server is a new feature to VI3, as in previous versions of ESX you would only have the host license mode. Take a look here and here (starting at page 33) to have a clearer picture on how to activate your licenses, generate license files and configure the License Server.

Installing VMware ESX3 Server

With VC2 and License Server all in place, finally dedicated to setup ESX 3 on the xSeries box. Again, for this matter, this manual can become very useful.

Configured partitions according to the following list:

/boot 250mb
/swp 1600mb
/ 8000mb
/tmp 4096mb
/home 4096mb
/var/log 2000mb
/vmfs 14572mb

Remember I'm not relying on internal storage to allocate VMFS partitions, hence the relatively small 14Gb /vmfs partition.

Once the installation is over, activated all the applicable licences from VC2. Also configured the storage as follows:

Localvmfs vmhba2:0:0:7 14GB
vmfsc1 vmbha0:0:1:1 101.50GB
vmfsc2 vmbha0:0:2:1 1.91TB
vmfsc3 vmbha0:0:3:1 1.09TB

NFS

On VC2, tried to configure the Snap Appliances as mounted NFS volumes, but an old version of Guardian OS (2.5) prevented me to use NFS3 via TCP. Yes, unfortunately, VC2 will only add NFS volumes as datastores if using NFS3 via TCP. You can still mount NFS volumes at the COS, not as datastores though. Open the outgoing NFS traffic on ESX3 firewall by entering the command

esxcfg-firewall -e nfsClient

Then mount the NFS volume using the command

mount -t nfs nfs_host:/share mounting_point

Networking

On VC2, registered the host and configured the network interfaces. As the xSeries host was only having 2 interfaces (1 x Gbit and 1 x 10/100Mbit), configured the Gbit interface to be used by VMs and by VMkernel (remember Vmotion needs a Gbit interface) and configured the second one to be used by Service Console.

Make sure all needed features are licensed. Remember that HA and DRS can only be applied to hosts that are part of a VC2 cluster.

Here are some last tips on the ESX configuration that are always handy:

SSH

SSH to the ESX3 is disabled and not allowed to the root user. You'll need to edit the file /etc/ssh/sshd_config and change the line PermitRootLogin no to PermitRootLogin yes

Because this service is not allowed on ESX3 firewall, if you want to SSH to other systems, you'll have to allow SSH outgoing traffic issuing the command

esxcfg-firewall -e sshClient

Tip: There's a free utility called Putty very useful to access your host via SSH.

FTP

Contrary to previous ESX versions, FTP server on port 21 is not present anymore. For security reasons, on ESX3 you can only be accessed using SFTP.

VM Templates

With the ESX3 box up and running, created the necessary templates. This is a very simple process that can save hours of work by significantly reduce new VMs deployment time. Simply start a new VM, attaching the right ISO file as the CD ROM drive and install the pretended operating system. I made 2003 Standard and Enterprise Edition servers. Proceed with the installation as usual. Once its finished, make the relevant modifications, install VMware tools and apply all updates and fixes availables and sysprep the server. Once the whole thing is done, on VC2 right click on the VM and either pick the option Clone to Template (if you still wish to use the VM) or Convert to Template (the VM will no longer exist as such, and it will be converted on a template). Please note that the template won't show on the 'Hosts and Clusters' inventory view. You'll have to switch to the 'Virtual Machines and Templates'.

Backup solution

Configurations on DR site are pretty much done. Time to install the backup software on the live ESX hosts.

Remember that this project is running on an extremely tight budget (if any!). In an ideal solution, I'd consider a link upgrade, usage of fiber and LightSand devices to interconect the SANs (or at the very least, a replication solution like Double-Take) and a more enterprise oriented backup application like esxRanger from Vizioncore.

Tip: There's a free, yet good, utility called WinSCP to copy your VMDKs directly from the ESX host to your Windows workstation.

Back do the backup solution. Aiming for a free solution for cost reasons, I first tried ESXpress because of it's delta technology backups and VBA (Virtual Backup Appliances). Because it only uses plain FTP as repository, I couldn't use:

- the Snap Appliances because the old Guardian OS 2.5 will not handle files bigger than 2Gb and because ESXpress sends the whole VMDK file (as opposed to some competitor products that export the VMDK files therefore dividing them into 2Gb chunks) which often would be larger than 2Gb.

- the ESX host at the DR site, as ESX3 does not use plain FTP server (instead it uses SFTP)

I ended up using a commonly used tool kown as VMBK.PL.

It would be quicker and far more practical if there was a way of backing up the VMs directly to the VMFS volumes on the SAN, but ESX3 cannot be set as FTP or NFS server, so backups will be made via NFS to the Snap Appliances.

The file /usr/local/bin/vmbk-default.conf (configuration file) looks like this (changes from default in red):

#Version 1.01.2
#set timeout
Timeout=60000
#minimum space required for add RedoLOG
minspaceforRedo=1024
# backup esx host configuration
backupESX=true

BackupSession="default"
#directory where file
destination=/mnt/vmbk/
#minimum space required for backup
minspace=1024
#minimum space required for add RedoLOG
minspaceforRedo=1000
#vmbk do not create subdirectory
FlatDir=false
#Create a restore shell script
Restore=true
# Backup configuration file and CMOS
BackupVMX=true
# the exported virtual disk contains the redo log if exist
BackREDO=false
#disk format VMDK or DSK
DiskFormat=VMDK
#Backup all Guest
BackupAllGuest=true
#format of log HTML or TEXT
LogFormat=HTML
#log file an directory with number of day in file name
logfile=/usr/lib/vmware-mui/apache/htdocs/vmbk_logs/log.html
#use cp command to disk instead of vmkfstools
usecp=true
#use vmkfstool(raw mode more faster than cp. !!!!caution with smb share!!!!) command to disk instead of cp
vmkfstoolsrawmode=false
#create a tar.gz file one for each disk inside a unique guest directory
usetar=false
#create a tar.gz one for each vm guest with inside all owned file
usetaronefile=false
#create a zip for each vm guest
usegzip=false

#tardir=/tmp not more used
# true or false
email=true
smtpserver=mail_server
to=name.surname@domain.com
from=name.surname@domain.com
encode=base64
html=true

# Force dismount before a mount operation
forcedismount=false

# Mount a nfs volume
nfs=false
nfsmount=host:/vol
nfsmountpoint=/vmbk
nfsoptions="soft"
# mount a samba volume
smb=false
smbserver=//smbserver/share
smbuser=domain\\user
smbpasswd=password
smbmountpoint=/vmbk
#smboptions=%none%

# true or false
ftp=false
ftpserver=server
ftpdir="/"
ftpuser=user
ftppasswd=password
# true or false
ftppassive=false
#ftptimeout value expressed in sec
ftptimeout=20

predirname=%none%
# predirname=%hostname%
# predirname=%date%
# predirname=%time%
# -N file create a list of backup files
# Create a list of file to backup
backupdisklist=false
backupdisklistfile=/tmp/vmbklist
# only do add redo (without redo commit)
onlyaddredo=false
# only do redo commit (without add redo)
onlyredocommit=false

runonstart=false
runonstartfile=""
runonstop=false
runonstopfile=""
#Veritas Netbackup Options
netbackup=false
netbackuppolicy="vmware"
netbackupclientname=%hostname%
netbackuplog="/var/log/vmbk_netbackup.log"
netbackupprepost=false
netbackupmaster="masterserver"
#Networker Legato Options
networker=false
networkerserver="server"
networkergroup="vmware"
networkerclientname=%hostname%
networkerlog="/var/log/vmbk_networker.log"

# Use temporary dir
temp=false
tempdir="/tmp"
#use syslog
syslog=true
facility="local6"
level="info"
#experimental
#redoredo=true

Mounted the Snap Appliance share as a NFS volume (see command line above).

The command line used to start the backup is:

vmbk.pl -x /home/vmware/VM_name/VM_name.vmx -C /usr/local/bin/vmbk-default.conf

-t does the whole procedure but in test mode (do not copy the VMDK files)
-x specifies a particular VM to be backed up
-C reads the command's parameters from the specified configuration file

Once all parameters to use and VMs to backup are identified, it's time to write a simple shell script that will be called by CRON at intended schedules.

Scheduling the backups

As I was not using any fancy state of the art backup application, I had to rely on VMware ESX own scheduling mechanisms. The cron.

Step 1 - Create a shell script called vmbk-cron.sh that actually starts the backup itself. The code will include the command line above applied to every significant VM. It should look something like this:

#!/bin/bash
/usr/local/bin/vmbk.pl -x /home/vmware/vm1/vm1.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm2/vm2.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm3/vm3.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm4/vm4.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul

Just keep appending a new line per new VM to backup. Also, if there's the need of having a VMBK log file per backup, make sure logging is not enabled on the vmbk-default.conf file, and instead use the following parameter for HTML files

-L /path/log_file.htm

or use this one for text files

-l /path/log_file.txt

added to the command lines. This will allow the creation of a secluded log file per backup operation.

Step 2 - Change the file's permission, in order to allow it's execution as a shell script file:

chmod 755 /usr/local/bin/vmbk-cron.sh

Step 3 - Test the script. At this point you might want to edit the script and and the -t switch to perform all backups in test mode (see above).

Step 4 - After confirming the operations' success, edit the file with the adequate scheduling parameters and copy it to /etc/cron.d/vmbk-cron.sh so that final version should look something like this:

#!/bin/bash
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm1/vm1.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm2/vm2.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm3/vm3.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm4/vm4.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul

The first 5 parameters of a CRON file entry will refer, respectively, to minute, hour, day, month and day of the week (0 = sunday). An asterisk means every.

This means the above script will backup VM1, VM2, VM3 and VM4 starting at 20:00 (8:00pm) on every saturday.

syslog

Once the right command line and scripts were in place, logging was needed. Because VMBK.PL has the possibility of sending messages via syslog, in case one wants to use it there might be a need for some extra configuration on the ESX host, like editing the file /etc/syslog.conf. For more information on syslog please read this older post.

Quick review on important configuration and log files

• The VMBK.PL logs are placed inside /usr/lib/vmware-mui/apache/htdocs/vmbk_logs/, so to access them either use the host's console or via web (https://host_name/vmbk_logs/log.html)

To be done:

build a backup shell script and insert in cron
build a shell script to copy VMDK files to the ESX host in order to reduce the total time of recovery from disaster processes documentation
configure VMs on ESX host pointing to VMDKs
build a shell script to startup all VMs

script to change network settings (10.1.x.x to 10.89.x.x)

I'll be more than happy to share information on this and other ESX3/DR projects with whoever asks for it. Just comment this post with your mail address and specific needs. Will try to reach everyone in a timely manner.