Select: Start -> Settings -> Network and Dial Up Connections
Select: Your Local Area Connection
Select: Internet Connections (TCP/IP) Properties
Change: Your IP Address and Subnet Mask and Gateway
Change: Preferred DNS server's address to the new server address
Select: OK -> OK -> Close
Your server's address is now changed.
Select: Start -> Programs -> Administrative Tools -> DNS
Double click Forward Look Up Zones. Then double click your server name.
Delete: Your type A records
Reset your information in your SOA and NS records and exit DNS.
Now drop down to an MS-DOS prompt and type the following:
ipconfig /flushdns Enter
Net Stop DNS Enter
Net Start DNS Enter
Net Stop Netlogon Enter
Net Start Netlogon Enter
ipconfig /registerdns Enter
You can now go back to DNS and make sure the records were all created and they should have changed the address to the correct address on their own.
Now run NSLookup from an MS-DOS prompt and see if all is resolved OK or not. It the names and IP addresses all resolve correctly, you're all set. If not, then go back to NSLookup and type
set root=servername
(the name of your DNS domain) and hit Enter. When the prompt returns, type
exit
to exit out of NSLookup. When you type NSLookup, it should resolve the name correctly.
Friday, 22 June 2007
Wednesday, 20 June 2007
ESX Setup: Defining block sizes
Attention!
When configuring ESX/creating new VMFS volumes, some people tend to mess around with block size. This will interfere with the maximum file size you can create on those volumes.
For example, if you then need a bigger VMDK file, you will need to delete and recreate your VMFS partition with a larger block size!
Block Size = Maximum file size
1024 = 256Gb
2048 = 500Gb
4096 = 1000Gb
When configuring ESX/creating new VMFS volumes, some people tend to mess around with block size. This will interfere with the maximum file size you can create on those volumes.
For example, if you then need a bigger VMDK file, you will need to delete and recreate your VMFS partition with a larger block size!
Block Size = Maximum file size
1024 = 256Gb
2048 = 500Gb
4096 = 1000Gb
Tuesday, 19 June 2007
Microsoft's Script Repository
Tons of script samples form Microsoft. Here's one category I use alot: Configuring Network Settings
Monday, 18 June 2007
Scripts: Create VMs from a script
The following code is a shell script to create VMs. This particular one will create a virtual machine that has the following characteristics:
##### VM Creation Script #####################################
#Script Version 1.1
#Author David E. Hart
#Date 10-05-06
#
#--------+
# Purpose
#--------+-----------------------------------------------------
# This script will create a VM with the following attributes;
# Virtual Machine Name = ScriptedVM
# Location of Virtual Machine = /VMFS/volumes/storage1/ScriptedVM
# Virtual Machine Type = "Microsoft Windows 2003 Standard"
# Virtual Machine Memory Allocation = 256 meg
#
#----------------------------------------+
#Custom Variable Section for Modification
#----------------------------------------+---------------------
#NVM is name of virtual machine(NVM). No Spaces allowed in name
#NVMDIR is the directory which holds all the VM files
#NVMOS specifies VM Operating System
#NVMSIZE is the size of the virtual disk to be created
#--------------------------------------------------------------
###############################################################
### Default Variable settings - change this to your preferences
NVM="ScriptedVM" # Name of Virtual Machine
NVMDIR="ScriptedVM" # Specify only the folder name to be created; NOT the complete path
NVMOS="winnetstandard" # Type of OS for Virtual Machine
NVMSIZE="4g" # Size of Virtual Machine Disk
VMMEMSIZE="256" # Default Memory Size
### End Variable Declaration
mkdir /vmfs/volumes/storage1/$NVMDIR # Creates directory
exec 6>&1 # Sets up write to file
exec 1>/vmfs/volumes/storage1/$NVMDIR/$NVM.vmx # Open file
# write the configuration
echo config.version = '"'6'"' # For ESX 3.x the value is 8
echo virtualHW.version = '"'3'"' # For ESX 3.x the value is 4
echo memsize = '"'$VMMEMSIZE'"'
www.syngress.com
Building a VM • Chapter 4 151
370_VMware_Tools_04_dummy.qxd 10/12/06 7:28 PM Page 151
echo floppy0.present = '"'TRUE'"' # setup VM with floppy
echo displayName = '"'$NVM'"' # name of virtual machine
echo guestOS = '"'$NVMOS'"'
echo
echo ide0:0.present = '"'TRUE'"'
echo ide0:0.deviceType = '"'cdrom-raw'"'
echo ide:0.startConnected = '"'false'"' # CDROM enabled
echo floppy0.startConnected = '"'FALSE'"'
echo floppy0.fileName = '"'/dev/fd0'"'
echo Ethernet0.present = '"'TRUE'"'
echo Ethernet0.networkName = '"'VM Network'"' # Default network
echo Ethernet0.addressType = '"'vpx'"'
echo
echo scsi0.present = '"'true'"'
echo scsi0.sharedBus = '"'none'"'
echo scsi0.virtualDev = '"'lsilogic'"'
echo scsi0:0.present = '"'true'"' # Virtual Disk Settings
echo scsi0:0.fileName = '"'$NVM.vmdk'"'
echo scsi0:0.deviceType = '"'scsi-hardDisk'"'
echo
# close file
exec 1>&-
# make stdout a copy of FD 6 (reset stdout), and close FD6
exec 1>&6
exec 6>&-
# Change permissions on the file so it can be executed by anyone
chmod 755 /vmfs/volumes/storage1/$NVMDIR/$NVM.vmx
#Creates 4gb Virtual disk
cd /vmfs/volumes/storage1/$NVMDIR #change to the VM dir
vmkfstools -c $NVMSIZE $NVM.vmdk -a lsilogic
#Register VM
vmware-cmd -s register /vmfs/volumes/storage1/$NVMDIR/$NVM.vmx
- VM called ScriptedVM in a directory named ScriptedVM on storage1
- assigned 256MB of memory
- will have a 4GB SCSI hard drive (lsilogic controller)
- configured for a Windows 2003 standard operating system
- floppy drive assigned, not connected at startup
- CD-ROM attached to the ESX server's CD-ROM drive, not connected at startup
- Ethernet adapter connected to the VM Network, enabled at startup
##### VM Creation Script #####################################
#Script Version 1.1
#Author David E. Hart
#Date 10-05-06
#
#--------+
# Purpose
#--------+-----------------------------------------------------
# This script will create a VM with the following attributes;
# Virtual Machine Name = ScriptedVM
# Location of Virtual Machine = /VMFS/volumes/storage1/ScriptedVM
# Virtual Machine Type = "Microsoft Windows 2003 Standard"
# Virtual Machine Memory Allocation = 256 meg
#
#----------------------------------------+
#Custom Variable Section for Modification
#----------------------------------------+---------------------
#NVM is name of virtual machine(NVM). No Spaces allowed in name
#NVMDIR is the directory which holds all the VM files
#NVMOS specifies VM Operating System
#NVMSIZE is the size of the virtual disk to be created
#--------------------------------------------------------------
###############################################################
### Default Variable settings - change this to your preferences
NVM="ScriptedVM" # Name of Virtual Machine
NVMDIR="ScriptedVM" # Specify only the folder name to be created; NOT the complete path
NVMOS="winnetstandard" # Type of OS for Virtual Machine
NVMSIZE="4g" # Size of Virtual Machine Disk
VMMEMSIZE="256" # Default Memory Size
### End Variable Declaration
mkdir /vmfs/volumes/storage1/$NVMDIR # Creates directory
exec 6>&1 # Sets up write to file
exec 1>/vmfs/volumes/storage1/$NVMDIR/$NVM.vmx # Open file
# write the configuration
echo config.version = '"'6'"' # For ESX 3.x the value is 8
echo virtualHW.version = '"'3'"' # For ESX 3.x the value is 4
echo memsize = '"'$VMMEMSIZE'"'
www.syngress.com
Building a VM • Chapter 4 151
370_VMware_Tools_04_dummy.qxd 10/12/06 7:28 PM Page 151
echo floppy0.present = '"'TRUE'"' # setup VM with floppy
echo displayName = '"'$NVM'"' # name of virtual machine
echo guestOS = '"'$NVMOS'"'
echo
echo ide0:0.present = '"'TRUE'"'
echo ide0:0.deviceType = '"'cdrom-raw'"'
echo ide:0.startConnected = '"'false'"' # CDROM enabled
echo floppy0.startConnected = '"'FALSE'"'
echo floppy0.fileName = '"'/dev/fd0'"'
echo Ethernet0.present = '"'TRUE'"'
echo Ethernet0.networkName = '"'VM Network'"' # Default network
echo Ethernet0.addressType = '"'vpx'"'
echo
echo scsi0.present = '"'true'"'
echo scsi0.sharedBus = '"'none'"'
echo scsi0.virtualDev = '"'lsilogic'"'
echo scsi0:0.present = '"'true'"' # Virtual Disk Settings
echo scsi0:0.fileName = '"'$NVM.vmdk'"'
echo scsi0:0.deviceType = '"'scsi-hardDisk'"'
echo
# close file
exec 1>&-
# make stdout a copy of FD 6 (reset stdout), and close FD6
exec 1>&6
exec 6>&-
# Change permissions on the file so it can be executed by anyone
chmod 755 /vmfs/volumes/storage1/$NVMDIR/$NVM.vmx
#Creates 4gb Virtual disk
cd /vmfs/volumes/storage1/$NVMDIR #change to the VM dir
vmkfstools -c $NVMSIZE $NVM.vmdk -a lsilogic
#Register VM
vmware-cmd -s register /vmfs/volumes/storage1/$NVMDIR/$NVM.vmx
Saturday, 16 June 2007
VI3, Disaster Recovery and Business Continuity
If there's one big advantage of using VI3, it certainly is its ability to ensure business continuity. Previous versions have done a great job, but VI3 and its new features like Vmware Consolidated Backup (VCB), High Availability (HA) and Dynamic Resource Scheduler (DRS) are really pushing it one step further.
The problem is although technology is making it easier every day, mentality is still an issue. The company I'm currently working for is just another example of it.
How it should be
We all know DR/BCP projects should be oriented to business needs and expectations, that have to be identified, studied, agreed and documented. Platform's RPOs and RTOs have to be investigated. Costs for outages have to be calculated. Risks have to be identified and mitigated. Reliability for the whole project has to be tested and assessed.
How it is sometimes
Now my current project is going nowhere near that way. For budget, time and political reasons my current DR project is being managed exactly the other way around: "Here is the available infrastructure and applications. Come up with the best DR/BC solution you can."
At the DR site (at least I was given one) I'm currently working at the infrastructure level, and according to my project plan I'll be finishing by the end of this month. With what I was provided (so far) my intentions are to take advantage out of the Vmware 2.5.x already implemented infrastructure, recover the critical platforms to this site and assume a "business as usual" status 12 hours after disaster situation is declared.
Scenario
I should now provide a high level picture of what I have. My main production site is some 300 kms away which I'm connected to via a 10Mbit circuit.
I also lucky enough to have:
- an IBM xSeries (with 4 x Xeon 2.5GHz, 8Gb RAM and a 34Gb RAID5 volume)
- an already working SAN (old IBM FastT700 - renamed to DS4000 - and 2 x Brocade/IBM 16 port fiber switch)
- a bunch of Snap Appliances (models 4200 and 2200)
I started by rack-mounting everything followed by passing all network and fiber cables. Connected the console. Added 8 more 300Gb hard drives (making a total of 12) to the EXP enclosures and 4 x 34Gb old hard drives that I have left.
Storage
After wiping all the existing information on the SAN and disks, used IBM Storage Manager 9 to reconfigure all adapters, WWNs, groups, hosts and ports. Also configured 3 brand new RAID 5 arrays.
Each array holds one LUN ans was configured as follows:
Array 1 - made with the old 34Gb disks, has one hot spare disk, and it will hold less critical data like images and ISOs (total: 101,5Gb)
Array 2 - made with the 8 new 300Gb disks, shares one hot spare disk with array 3 and it will hold backups of critical VMs, ready to be started up (total: 1,9Tb)
Array 3 - made with the last 300Gb disks, will also hold VMs (total: 1.1Tb)
VirtualCentre 2 and License Server
I then provisioned an HP ML series server (with 2 x P4 2.7GHz, 2Gb of RAM and a total of 101Gb). This was installed with the usual HP tools and MS Windows 2003 Standard Edition. All updates and fixes were applied. Finally installed VirtualCentre 2 and License Server.
Note: The License server is a new feature to VI3, as in previous versions of ESX you would only have the host license mode. Take a look here and here (starting at page 33) to have a clearer picture on how to activate your licenses, generate license files and configure the License Server.
Installing VMware ESX3 Server
With VC2 and License Server all in place, finally dedicated to setup ESX 3 on the xSeries box. Again, for this matter, this manual can become very useful.
Configured partitions according to the following list:
/boot 250mb
/swp 1600mb
/ 8000mb
/tmp 4096mb
/home 4096mb
/var/log 2000mb
/vmfs 14572mb
Remember I'm not relying on internal storage to allocate VMFS partitions, hence the relatively small 14Gb /vmfs partition.
Once the installation is over, activated all the applicable licences from VC2. Also configured the storage as follows:
Localvmfs vmhba2:0:0:7 14GB
vmfsc1 vmbha0:0:1:1 101.50GB
vmfsc2 vmbha0:0:2:1 1.91TB
vmfsc3 vmbha0:0:3:1 1.09TB
NFS
On VC2, tried to configure the Snap Appliances as mounted NFS volumes, but an old version of Guardian OS (2.5) prevented me to use NFS3 via TCP. Yes, unfortunately, VC2 will only add NFS volumes as datastores if using NFS3 via TCP. You can still mount NFS volumes at the COS, not as datastores though. Open the outgoing NFS traffic on ESX3 firewall by entering the command
esxcfg-firewall -e nfsClient
Then mount the NFS volume using the command
mount -t nfs nfs_host:/share mounting_point
Networking
On VC2, registered the host and configured the network interfaces. As the xSeries host was only having 2 interfaces (1 x Gbit and 1 x 10/100Mbit), configured the Gbit interface to be used by VMs and by VMkernel (remember Vmotion needs a Gbit interface) and configured the second one to be used by Service Console.
Make sure all needed features are licensed. Remember that HA and DRS can only be applied to hosts that are part of a VC2 cluster.
Here are some last tips on the ESX configuration that are always handy:
SSH
SSH to the ESX3 is disabled and not allowed to the root user. You'll need to edit the file /etc/ssh/sshd_config and change the line PermitRootLogin no to PermitRootLogin yes
Because this service is not allowed on ESX3 firewall, if you want to SSH to other systems, you'll have to allow SSH outgoing traffic issuing the command
esxcfg-firewall -e sshClient
Tip: There's a free utility called Putty very useful to access your host via SSH.
FTP
Contrary to previous ESX versions, FTP server on port 21 is not present anymore. For security reasons, on ESX3 you can only be accessed using SFTP.
VM Templates
With the ESX3 box up and running, created the necessary templates. This is a very simple process that can save hours of work by significantly reduce new VMs deployment time. Simply start a new VM, attaching the right ISO file as the CD ROM drive and install the pretended operating system. I made 2003 Standard and Enterprise Edition servers. Proceed with the installation as usual. Once its finished, make the relevant modifications, install VMware tools and apply all updates and fixes availables and sysprep the server. Once the whole thing is done, on VC2 right click on the VM and either pick the option Clone to Template (if you still wish to use the VM) or Convert to Template (the VM will no longer exist as such, and it will be converted on a template). Please note that the template won't show on the 'Hosts and Clusters' inventory view. You'll have to switch to the 'Virtual Machines and Templates'.
Backup solution
Configurations on DR site are pretty much done. Time to install the backup software on the live ESX hosts.
Remember that this project is running on an extremely tight budget (if any!). In an ideal solution, I'd consider a link upgrade, usage of fiber and LightSand devices to interconect the SANs (or at the very least, a replication solution like Double-Take) and a more enterprise oriented backup application like esxRanger from Vizioncore.
Tip: There's a free, yet good, utility called WinSCP to copy your VMDKs directly from the ESX host to your Windows workstation.
Back do the backup solution. Aiming for a free solution for cost reasons, I first tried ESXpress because of it's delta technology backups and VBA (Virtual Backup Appliances). Because it only uses plain FTP as repository, I couldn't use:
- the Snap Appliances because the old Guardian OS 2.5 will not handle files bigger than 2Gb and because ESXpress sends the whole VMDK file (as opposed to some competitor products that export the VMDK files therefore dividing them into 2Gb chunks) which often would be larger than 2Gb.
- the ESX host at the DR site, as ESX3 does not use plain FTP server (instead it uses SFTP)
I ended up using a commonly used tool kown as VMBK.PL.
It would be quicker and far more practical if there was a way of backing up the VMs directly to the VMFS volumes on the SAN, but ESX3 cannot be set as FTP or NFS server, so backups will be made via NFS to the Snap Appliances.
The file /usr/local/bin/vmbk-default.conf (configuration file) looks like this (changes from default in red):
#Version 1.01.2
#set timeout
Timeout=60000
#minimum space required for add RedoLOG
minspaceforRedo=1024
# backup esx host configuration
backupESX=true
BackupSession="default"
#directory where file
destination=/mnt/vmbk/
#minimum space required for backup
minspace=1024
#minimum space required for add RedoLOG
minspaceforRedo=1000
#vmbk do not create subdirectory
FlatDir=false
#Create a restore shell script
Restore=true
# Backup configuration file and CMOS
BackupVMX=true
# the exported virtual disk contains the redo log if exist
BackREDO=false
#disk format VMDK or DSK
DiskFormat=VMDK
#Backup all Guest
BackupAllGuest=true
#format of log HTML or TEXT
LogFormat=HTML
#log file an directory with number of day in file name
logfile=/usr/lib/vmware-mui/apache/htdocs/vmbk_logs/log.html
#use cp command to disk instead of vmkfstools
usecp=true
#use vmkfstool(raw mode more faster than cp. !!!!caution with smb share!!!!) command to disk instead of cp
vmkfstoolsrawmode=false
#create a tar.gz file one for each disk inside a unique guest directory
usetar=false
#create a tar.gz one for each vm guest with inside all owned file
usetaronefile=false
#create a zip for each vm guest
usegzip=false
#tardir=/tmp not more used
# true or false
email=true
smtpserver=mail_server
to=name.surname@domain.com
from=name.surname@domain.com
encode=base64
html=true
# Force dismount before a mount operation
forcedismount=false
# Mount a nfs volume
nfs=false
nfsmount=host:/vol
nfsmountpoint=/vmbk
nfsoptions="soft"
# mount a samba volume
smb=false
smbserver=//smbserver/share
smbuser=domain\\user
smbpasswd=password
smbmountpoint=/vmbk
#smboptions=%none%
# true or false
ftp=false
ftpserver=server
ftpdir="/"
ftpuser=user
ftppasswd=password
# true or false
ftppassive=false
#ftptimeout value expressed in sec
ftptimeout=20
predirname=%none%
# predirname=%hostname%
# predirname=%date%
# predirname=%time%
# -N file create a list of backup files
# Create a list of file to backup
backupdisklist=false
backupdisklistfile=/tmp/vmbklist
# only do add redo (without redo commit)
onlyaddredo=false
# only do redo commit (without add redo)
onlyredocommit=false
runonstart=false
runonstartfile=""
runonstop=false
runonstopfile=""
#Veritas Netbackup Options
netbackup=false
netbackuppolicy="vmware"
netbackupclientname=%hostname%
netbackuplog="/var/log/vmbk_netbackup.log"
netbackupprepost=false
netbackupmaster="masterserver"
#Networker Legato Options
networker=false
networkerserver="server"
networkergroup="vmware"
networkerclientname=%hostname%
networkerlog="/var/log/vmbk_networker.log"
# Use temporary dir
temp=false
tempdir="/tmp"
#use syslog
syslog=true
facility="local6"
level="info"
#experimental
#redoredo=true
Mounted the Snap Appliance share as a NFS volume (see command line above).
The command line used to start the backup is:
vmbk.pl -x /home/vmware/VM_name/VM_name.vmx -C /usr/local/bin/vmbk-default.conf
-t does the whole procedure but in test mode (do not copy the VMDK files)
-x specifies a particular VM to be backed up
-C reads the command's parameters from the specified configuration file
Once all parameters to use and VMs to backup are identified, it's time to write a simple shell script that will be called by CRON at intended schedules.
Scheduling the backups
As I was not using any fancy state of the art backup application, I had to rely on VMware ESX own scheduling mechanisms. The cron.
Step 1 - Create a shell script called vmbk-cron.sh that actually starts the backup itself. The code will include the command line above applied to every significant VM. It should look something like this:
#!/bin/bash
/usr/local/bin/vmbk.pl -x /home/vmware/vm1/vm1.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm2/vm2.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm3/vm3.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm4/vm4.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
Just keep appending a new line per new VM to backup. Also, if there's the need of having a VMBK log file per backup, make sure logging is not enabled on the vmbk-default.conf file, and instead use the following parameter for HTML files
-L /path/log_file.htm
or use this one for text files
-l /path/log_file.txt
added to the command lines. This will allow the creation of a secluded log file per backup operation.
Step 2 - Change the file's permission, in order to allow it's execution as a shell script file:
chmod 755 /usr/local/bin/vmbk-cron.sh
Step 3 - Test the script. At this point you might want to edit the script and and the -t switch to perform all backups in test mode (see above).
Step 4 - After confirming the operations' success, edit the file with the adequate scheduling parameters and copy it to /etc/cron.d/vmbk-cron.sh so that final version should look something like this:
#!/bin/bash
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm1/vm1.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm2/vm2.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm3/vm3.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm4/vm4.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
The first 5 parameters of a CRON file entry will refer, respectively, to minute, hour, day, month and day of the week (0 = sunday). An asterisk means every.
This means the above script will backup VM1, VM2, VM3 and VM4 starting at 20:00 (8:00pm) on every saturday.
The problem is although technology is making it easier every day, mentality is still an issue. The company I'm currently working for is just another example of it.
How it should be
We all know DR/BCP projects should be oriented to business needs and expectations, that have to be identified, studied, agreed and documented. Platform's RPOs and RTOs have to be investigated. Costs for outages have to be calculated. Risks have to be identified and mitigated. Reliability for the whole project has to be tested and assessed.
How it is sometimes
Now my current project is going nowhere near that way. For budget, time and political reasons my current DR project is being managed exactly the other way around: "Here is the available infrastructure and applications. Come up with the best DR/BC solution you can."
At the DR site (at least I was given one) I'm currently working at the infrastructure level, and according to my project plan I'll be finishing by the end of this month. With what I was provided (so far) my intentions are to take advantage out of the Vmware 2.5.x already implemented infrastructure, recover the critical platforms to this site and assume a "business as usual" status 12 hours after disaster situation is declared.
Scenario
I should now provide a high level picture of what I have. My main production site is some 300 kms away which I'm connected to via a 10Mbit circuit.
I also lucky enough to have:
- an IBM xSeries (with 4 x Xeon 2.5GHz, 8Gb RAM and a 34Gb RAID5 volume)
- an already working SAN (old IBM FastT700 - renamed to DS4000 - and 2 x Brocade/IBM 16 port fiber switch)
- a bunch of Snap Appliances (models 4200 and 2200)
I started by rack-mounting everything followed by passing all network and fiber cables. Connected the console. Added 8 more 300Gb hard drives (making a total of 12) to the EXP enclosures and 4 x 34Gb old hard drives that I have left.
Storage
After wiping all the existing information on the SAN and disks, used IBM Storage Manager 9 to reconfigure all adapters, WWNs, groups, hosts and ports. Also configured 3 brand new RAID 5 arrays.
Each array holds one LUN ans was configured as follows:
Array 1 - made with the old 34Gb disks, has one hot spare disk, and it will hold less critical data like images and ISOs (total: 101,5Gb)
Array 2 - made with the 8 new 300Gb disks, shares one hot spare disk with array 3 and it will hold backups of critical VMs, ready to be started up (total: 1,9Tb)
Array 3 - made with the last 300Gb disks, will also hold VMs (total: 1.1Tb)
VirtualCentre 2 and License Server
I then provisioned an HP ML series server (with 2 x P4 2.7GHz, 2Gb of RAM and a total of 101Gb). This was installed with the usual HP tools and MS Windows 2003 Standard Edition. All updates and fixes were applied. Finally installed VirtualCentre 2 and License Server.
Note: The License server is a new feature to VI3, as in previous versions of ESX you would only have the host license mode. Take a look here and here (starting at page 33) to have a clearer picture on how to activate your licenses, generate license files and configure the License Server.
Installing VMware ESX3 Server
With VC2 and License Server all in place, finally dedicated to setup ESX 3 on the xSeries box. Again, for this matter, this manual can become very useful.
Configured partitions according to the following list:
/boot 250mb
/swp 1600mb
/ 8000mb
/tmp 4096mb
/home 4096mb
/var/log 2000mb
/vmfs 14572mb
Remember I'm not relying on internal storage to allocate VMFS partitions, hence the relatively small 14Gb /vmfs partition.
Once the installation is over, activated all the applicable licences from VC2. Also configured the storage as follows:
Localvmfs vmhba2:0:0:7 14GB
vmfsc1 vmbha0:0:1:1 101.50GB
vmfsc2 vmbha0:0:2:1 1.91TB
vmfsc3 vmbha0:0:3:1 1.09TB
NFS
On VC2, tried to configure the Snap Appliances as mounted NFS volumes, but an old version of Guardian OS (2.5) prevented me to use NFS3 via TCP. Yes, unfortunately, VC2 will only add NFS volumes as datastores if using NFS3 via TCP. You can still mount NFS volumes at the COS, not as datastores though. Open the outgoing NFS traffic on ESX3 firewall by entering the command
esxcfg-firewall -e nfsClient
Then mount the NFS volume using the command
mount -t nfs nfs_host
Networking
On VC2, registered the host and configured the network interfaces. As the xSeries host was only having 2 interfaces (1 x Gbit and 1 x 10/100Mbit), configured the Gbit interface to be used by VMs and by VMkernel (remember Vmotion needs a Gbit interface) and configured the second one to be used by Service Console.
Make sure all needed features are licensed. Remember that HA and DRS can only be applied to hosts that are part of a VC2 cluster.
Here are some last tips on the ESX configuration that are always handy:
SSH
SSH to the ESX3 is disabled and not allowed to the root user. You'll need to edit the file /etc/ssh/sshd_config and change the line PermitRootLogin no to PermitRootLogin yes
Because this service is not allowed on ESX3 firewall, if you want to SSH to other systems, you'll have to allow SSH outgoing traffic issuing the command
esxcfg-firewall -e sshClient
Tip: There's a free utility called Putty very useful to access your host via SSH.
FTP
Contrary to previous ESX versions, FTP server on port 21 is not present anymore. For security reasons, on ESX3 you can only be accessed using SFTP.
VM Templates
With the ESX3 box up and running, created the necessary templates. This is a very simple process that can save hours of work by significantly reduce new VMs deployment time. Simply start a new VM, attaching the right ISO file as the CD ROM drive and install the pretended operating system. I made 2003 Standard and Enterprise Edition servers. Proceed with the installation as usual. Once its finished, make the relevant modifications, install VMware tools and apply all updates and fixes availables and sysprep the server. Once the whole thing is done, on VC2 right click on the VM and either pick the option Clone to Template (if you still wish to use the VM) or Convert to Template (the VM will no longer exist as such, and it will be converted on a template). Please note that the template won't show on the 'Hosts and Clusters' inventory view. You'll have to switch to the 'Virtual Machines and Templates'.
Backup solution
Configurations on DR site are pretty much done. Time to install the backup software on the live ESX hosts.
Remember that this project is running on an extremely tight budget (if any!). In an ideal solution, I'd consider a link upgrade, usage of fiber and LightSand devices to interconect the SANs (or at the very least, a replication solution like Double-Take) and a more enterprise oriented backup application like esxRanger from Vizioncore.
Tip: There's a free, yet good, utility called WinSCP to copy your VMDKs directly from the ESX host to your Windows workstation.
Back do the backup solution. Aiming for a free solution for cost reasons, I first tried ESXpress because of it's delta technology backups and VBA (Virtual Backup Appliances). Because it only uses plain FTP as repository, I couldn't use:
- the Snap Appliances because the old Guardian OS 2.5 will not handle files bigger than 2Gb and because ESXpress sends the whole VMDK file (as opposed to some competitor products that export the VMDK files therefore dividing them into 2Gb chunks) which often would be larger than 2Gb.
- the ESX host at the DR site, as ESX3 does not use plain FTP server (instead it uses SFTP)
I ended up using a commonly used tool kown as VMBK.PL.
It would be quicker and far more practical if there was a way of backing up the VMs directly to the VMFS volumes on the SAN, but ESX3 cannot be set as FTP or NFS server, so backups will be made via NFS to the Snap Appliances.
The file /usr/local/bin/vmbk-default.conf (configuration file) looks like this (changes from default in red):
#Version 1.01.2
#set timeout
Timeout=60000
#minimum space required for add RedoLOG
minspaceforRedo=1024
# backup esx host configuration
backupESX=true
BackupSession="default"
#directory where file
destination=/mnt/vmbk/
#minimum space required for backup
minspace=1024
#minimum space required for add RedoLOG
minspaceforRedo=1000
#vmbk do not create subdirectory
FlatDir=false
#Create a restore shell script
Restore=true
# Backup configuration file and CMOS
BackupVMX=true
# the exported virtual disk contains the redo log if exist
BackREDO=false
#disk format VMDK or DSK
DiskFormat=VMDK
#Backup all Guest
BackupAllGuest=true
#format of log HTML or TEXT
LogFormat=HTML
#log file an directory with number of day in file name
logfile=/usr/lib/vmware-mui/apache/htdocs/vmbk_logs/log.html
#use cp command to disk instead of vmkfstools
usecp=true
#use vmkfstool(raw mode more faster than cp. !!!!caution with smb share!!!!) command to disk instead of cp
vmkfstoolsrawmode=false
#create a tar.gz file one for each disk inside a unique guest directory
usetar=false
#create a tar.gz one for each vm guest with inside all owned file
usetaronefile=false
#create a zip for each vm guest
usegzip=false
#tardir=/tmp not more used
# true or false
email=true
smtpserver=mail_server
to=name.surname@domain.com
from=name.surname@domain.com
encode=base64
html=true
# Force dismount before a mount operation
forcedismount=false
# Mount a nfs volume
nfs=false
nfsmount=host:/vol
nfsmountpoint=/vmbk
nfsoptions="soft"
# mount a samba volume
smb=false
smbserver=//smbserver/share
smbuser=domain\\user
smbpasswd=password
smbmountpoint=/vmbk
#smboptions=%none%
# true or false
ftp=false
ftpserver=server
ftpdir="/"
ftpuser=user
ftppasswd=password
# true or false
ftppassive=false
#ftptimeout value expressed in sec
ftptimeout=20
predirname=%none%
# predirname=%hostname%
# predirname=%date%
# predirname=%time%
# -N file create a list of backup files
# Create a list of file to backup
backupdisklist=false
backupdisklistfile=/tmp/vmbklist
# only do add redo (without redo commit)
onlyaddredo=false
# only do redo commit (without add redo)
onlyredocommit=false
runonstart=false
runonstartfile=""
runonstop=false
runonstopfile=""
#Veritas Netbackup Options
netbackup=false
netbackuppolicy="vmware"
netbackupclientname=%hostname%
netbackuplog="/var/log/vmbk_netbackup.log"
netbackupprepost=false
netbackupmaster="masterserver"
#Networker Legato Options
networker=false
networkerserver="server"
networkergroup="vmware"
networkerclientname=%hostname%
networkerlog="/var/log/vmbk_networker.log"
# Use temporary dir
temp=false
tempdir="/tmp"
#use syslog
syslog=true
facility="local6"
level="info"
#experimental
#redoredo=true
Mounted the Snap Appliance share as a NFS volume (see command line above).
The command line used to start the backup is:
vmbk.pl -x /home/vmware/VM_name/VM_name.vmx -C /usr/local/bin/vmbk-default.conf
-t does the whole procedure but in test mode (do not copy the VMDK files)
-x specifies a particular VM to be backed up
-C reads the command's parameters from the specified configuration file
Once all parameters to use and VMs to backup are identified, it's time to write a simple shell script that will be called by CRON at intended schedules.
Scheduling the backups
As I was not using any fancy state of the art backup application, I had to rely on VMware ESX own scheduling mechanisms. The cron.
Step 1 - Create a shell script called vmbk-cron.sh that actually starts the backup itself. The code will include the command line above applied to every significant VM. It should look something like this:
#!/bin/bash
/usr/local/bin/vmbk.pl -x /home/vmware/vm1/vm1.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm2/vm2.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm3/vm3.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
/usr/local/bin/vmbk.pl -x /home/vmware/vm4/vm4.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
Just keep appending a new line per new VM to backup. Also, if there's the need of having a VMBK log file per backup, make sure logging is not enabled on the vmbk-default.conf file, and instead use the following parameter for HTML files
-L /path/log_file.htm
or use this one for text files
-l /path/log_file.txt
added to the command lines. This will allow the creation of a secluded log file per backup operation.
Step 2 - Change the file's permission, in order to allow it's execution as a shell script file:
chmod 755 /usr/local/bin/vmbk-cron.sh
Step 3 - Test the script. At this point you might want to edit the script and and the -t switch to perform all backups in test mode (see above).
Step 4 - After confirming the operations' success, edit the file with the adequate scheduling parameters and copy it to /etc/cron.d/vmbk-cron.sh so that final version should look something like this:
#!/bin/bash
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm1/vm1.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm2/vm2.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm3/vm3.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
00 20 * * 6 root /usr/local/bin/vmbk.pl -x /home/vmware/vm4/vm4.vmx -C /usr/local/bin/vmbk-default.conf > /dev/nul
The first 5 parameters of a CRON file entry will refer, respectively, to minute, hour, day, month and day of the week (0 = sunday). An asterisk means every.
This means the above script will backup VM1, VM2, VM3 and VM4 starting at 20:00 (8:00pm) on every saturday.
syslog
Once the right command line and scripts were in place, logging was needed. Because VMBK.PL has the possibility of sending messages via syslog, in case one wants to use it there might be a need for some extra configuration on the ESX host, like editing the file /etc/syslog.conf. For more information on syslog please read this older post.
Quick review on important configuration and log files
- The VMBK.PL logs are placed inside /usr/lib/vmware-mui/apache/htdocs/vmbk_logs/, so to access them either use the host's console or via web (https://host_name/vmbk_logs/log.html)
To be done:
build a backup shell script and insert in cron
build a shell script to copy VMDK files to the ESX host in order to reduce the total time of recovery from disaster processes documentation
configure VMs on ESX host pointing to VMDKs
build a shell script to startup all VMs
script to change network settings (10.1.x.x to 10.89.x.x)
I'll be more than happy to share information on this and other ESX3/DR projects with whoever asks for it. Just comment this post with your mail address and specific needs. Will try to reach everyone in a timely manner.
Top 10 Vmware lists, tips and howto's
Very nice top 10 Vmware lists, compiled by Eric Siebert:
- Top 10 things you must read if you are new to VMware
- Top 10 things you must read to get more technical with VI3
- Top 10 things you must read about Vmware Storage (SAN/NAS/iSCSI/NFS)
- Top 10 things you must read about Vmware Security
- Top 12 things you must read about Vmware Troubleshooting
- Top 10 things you must read about Vmware Networking
- Top 10 things you must read about VirtualCenter
- Top 11 things you must read about Vmware Performance
- Top 10 things you must read about Memory
Also checkout his:
- (claimed) largest VMware link collection on the planet
- list of Vmware tips and howto's
- and Converter tips
Well done Eric!
Linux (and friends) online resources
Here are some links to nice online resources, mostly books:
JAVA
Java in a Nutshell
Java Language Reference
Java AWT Reference
Java Fundamental Classes Reference
Exploring Java
Perl
Perl in a Nutshell
Learning Perl
Learning Perl on Win32 Systems
Programming Perl
Advanced Perl Programming
Perl Cookbook
Networking
DNS & BIND
TCP/IP Network Administration
sendmailsendmail Desktop Reference
Building Internet Firewalls
Practical UNIX & Internet Security
UNIX
UNIX Power Tools
UNIX in a Nutshell: System V Edition
Learning the vi Editorsed & awk
Learning the Korn Shell
Learning the UNIX Operating System
WWW
HTML: The Definitive Guide
CGI Programming on the World Wide Web
JavaScript: The Definitive Guide
Programming Perl
Web Master in a Nutshell
Others
Using Samba
JAVA
Java in a Nutshell
Java Language Reference
Java AWT Reference
Java Fundamental Classes Reference
Exploring Java
Perl
Perl in a Nutshell
Learning Perl
Learning Perl on Win32 Systems
Programming Perl
Advanced Perl Programming
Perl Cookbook
Networking
DNS & BIND
TCP/IP Network Administration
sendmailsendmail Desktop Reference
Building Internet Firewalls
Practical UNIX & Internet Security
UNIX
UNIX Power Tools
UNIX in a Nutshell: System V Edition
Learning the vi Editorsed & awk
Learning the Korn Shell
Learning the UNIX Operating System
WWW
HTML: The Definitive Guide
CGI Programming on the World Wide Web
JavaScript: The Definitive Guide
Programming Perl
Web Master in a Nutshell
Others
Using Samba
Shrinking and expanding disks with Vmware ESX
Some of ESX3 New Commands
Part of this article has been compiled by B2V consultants & trainers and is based upon their personal experiences with the VMware ESX Server 3 product and is updated frequently. (Check the full article at http://b2v.co.uk/content/view/31/42/)
This will briefly discuss on some commands new to ESX3.
esxcfg-
This is a new unified tool that is used to configure a large number of items in ESX Server 3.0.
esxcfg-advcfg
The esxcfg-advcfg command is interesting as there is not a huge amount of help about this command. However, we can figure out that it is meant to do advanced configuration and we can figure out some settings that can be made. The -g switch is used to "get" settings; the -s switch is used to "set" settings.
[root@esx1 vmware]# esxcfg-advcfg -g /Misc/BlueScreenTimeout
Value of BlueScreenTimeout is 0
[root@esx1 vmware]# esxcfg-advcfg -g /Misc/HostName
Value of HostName is esx1.vmlab.net
[root@esx1 vmware]# esxcfg-advcfg -g /VMFS3/ZeroedThickVirtualDisks
Value of ZeroedThickVirtualDisks is 1
[root@esx1 vmware]# esxcfg-advcfg –g /Disk/SupportSparseLUN
Value of SupportSparseLUN is 1
The question is, how much is configurable? To figure out what is configurable, we recommend that you look in the directory /proc/vmware/config in the service console and you will see the directories :
BufferCache
Cpu
Disk
FileSystem
Irq
LVM
Mem
Migrate
Misc
Net
NFS
Numa
Scsi
User
VMFS3
From these directories and the files within, you can work out the paths to be supplied to the esxcfg-advcfg command as parameters. Remember case sensitivity.
Usage: esxcfg-advcfg []
-g--get Get the value of the config option
-s--set Set the value of the config option
-d--default Reset Config option to default
-q--quiet Suppress output
-k--set-kernel Set a VMkernel load time option value.
-j--get-kernel Get a VMkernel load time option value.
-h--help Show this message.
-r--restore Restore all advanced options from the configuration file. (FOR INTERNAL USE ONLY).
esxcfg-firewall
The service console in ESX 3 now has a firewall enabled by default. We use this command to view and configure the firewall rules.
The most popular switch will be the -q switch to query the firewall for its settings.
[root@esxhost1 root]# esxcfg-firewall -q
The -s switch will allow you to enable or disable network services that may traverse the firewall successfully. The list of known services are shown below - very case sensitive!....
nfsClient
ftpServer
ntpClient
dellom
nisClient
vncServer
tmpLicenseClient
swISCSIClient
CIMHttpsServer
sshClient
snmpd
tmpAAMClient
vpxHeartbeats
smbClient
hpim
tmpHostVmdbServer
tmpHostdSOAPServer
ftpClient
sshServer
ibmdirector
CIMHttpServer
telnetClient
The -q switch queries the state of a particular known service.
The -l switch loads the firewall and enables the IP tables.
The -u switch unloads the firewall and disables the IP tables.
We use the -e switch to enable a particular known service, so if we wanted to enable ssh outbound connections from the service console we would simply enter
[root@esxhost1 root]# esxcfg-firewall -e sshClient
We use the -d switch to disable a service. In the following example, we prevent outbound connections
[root@esxhost1 root]# esxcfg-firewall -d smbClient
esxcfg-module
This command produces an output similar to vmkload_mod -list
[root@lithium06 tools-isoimages]# esxcfg-module -l
Module Type Enabled Loaded
vmkapimod vmkapimod true true
vmklinux linux true true
cciss.o scsi true false
tg3.o nic true false
qla2300_7xx.o fc true false
Although if you compare the output with the old command, things don't exactly match up. Not sure why just yet....
esxcfg-rescan
As vmkfstools -rescan
esxcfg-upgrade
esxcfg-upgrade -h --help
-g --convert-grub
-f --convert-fstab
-r --upgrade-pre-vmkernel
-o --upgrade-post-vmkernel
The -g option may only be used with the -r option.
esxcfg-vswitch
This command allows you to list, add, modify or delete virtual Ethernet switches on an ESX host. The simplest option with this command is the -l option to list the virtual switches defined on the host.
[root@esx root]# esxcfg-vswitch -l
If you are having problems with your ESX server after an in-place upgrade, this tool is invaluable in resolving the problems with service console networking.
The output of this command is initially a little intimidating. It is best to keep in mind the network topology:
Service Console IP Interface (vswif0) ---- connected to ----> Service Console Port on vSwitch
----- uplinked to ----> vmnic
Where a vmnic is a physical Ethernet adapter in the ESX server.
To add a virtual Ethernet switch, we use "-a" with the command
[root@esx root]# esxcfg-vswitch -a vSwitch3
esxcfg-auth
Configures the service console authentication options including NIS, LDAP, Kerberos and Active Directory.
esxcfg-info
Produces an enormous amount of information about the ESX host. You really need to pipe this to a file for closer examination!
[root@esx root]# esxcfg-info >esxinfo.txt
esxcfg-mpath
Manages multi-pathing just as the vmkmultipath utility did in previous versions of ESX Server.
[root@lithium06 tools-isoimages]# esxcfg-mpath -l
Disk vmhba0:0:0 /dev/cciss/c0d0 (69459MB) has 1 paths and policy of Fixed
Local 2:1.0 vmhba0:0:0 On active preferred
Disk vmhba1:0:0 (0MB) has 1 paths and policy of Most Recently Used
FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:0
On active preferred
Disk vmhba1:0:6 /dev/sda (9216MB) has 1 paths and policy of Most Recently Used
FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:6
On active preferred
Disk vmhba1:0:21 /dev/sdb (10240MB) has 1 paths and policy of Most Recently Used
FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:21
On active preferred
esxcfg-resgrp
Used to manage the new ESX feature called resource groups. This command can add, remove or modify existing resource groups.
esxcfg-hbadevs
The esxcfg-vmhbadevs command is used to list the equivalent Linux device names for the visible disk devices that the VMkernel references using vmhba notation.
[root@esx1 root]# esxcfg-vmhbadevs
vmhba0:0:0 /dev/sda vmhba0:0:1 /dev/sdb
vmhba0:0:2 /dev/sdcvmhba0:0:3 /dev/sdd
vmhba2:0:0 /dev/sdevmhba2:1:0 /dev/sdf
If we use this command with the –m switch, then we only list the LUNs which contain VMFS partitions. Alongside the Linux device name, a long unique hexadecimal value is listed. This is the VMFS volume signature assigned by the new logical volume manager (LVM).
[root@esx1 root]# esxcfg-vmhbadevs -m
vmhba0:0:0:1 /dev/sda1 45407607-fbc43ced-94cb-00145e231ce3
vmhba0:0:2:1 /dev/sdc1 455b08a8-8af7fee3-daa9-00145e231e35
vmhba2:0:0:3 /dev/sde3 4559c75f-831d8f3e-bc81-00145e231e35
You can view these volumes in the directory /vmfs/volumes/
esxcfg-boot
Used to configure the GRUB options presented at boot time. One thing to note is that the new esxcfg commands will not run if you boot just into Linux. If you just want to query the boot settings, you can use the -q switch but this must be qualified with the keyword boot or vmkmod.
[root@lithium06 tools-isoimages]# esxcfg-boot -q boot
272 2:;7:;10:; UUID=847199e4-d3c7-11da-8ef8-930e3d734c03 /vmlinuz-2.4.21-37.0.2.ELvmnix /initrd-2.4.21-37.0.2.ELvmnix.img
[root@lithium06 tools-isoimages]# esxcfg-boot -q vmkmod
vmkapimod vmkapimod
vmklinux linux
cciss.o scsi
tg3.o nic
qla2300_7xx.o fc
Not yet tested other options....
esxcfg-init
Should not be run manually!
esxcfg-nas
Used to configure access to Network Attached Storage (NAS).
esxcfg-nas [
-a--add Add a new NAS filesystem to /vmfs volumes. Requires --host and --share options.
-o--host Set the host name or ip address for a NAS mount.
-s--share Set the name of the NAS share on the remote system.
-d--delete Unmount and delete a filesystem.
-l--list List the currently mounted NAS file systems.
-r--restore Restore all NAS mounts from the configuration file. (FOR INTERNAL USE ONLY).
-h--help Show this message.
esxcfg-route
If we add an IP address to the VMkernel by adding a VMkernel port, then we can fully configure that IP stack by also assigning a default gateway. We can view (no parameters) and set (1st parameter) the VMkernel IP default gateway with the esxcfg-route command as shown here.
[root@esx1 etc]# esxcfg-route
VMkernel default gateway is 100.100.100.254
[root@esx1 etc]# esxcfg-route 100.100.100.1
VMkernel default gateway set to 100.100.100.1
esxcfg-vmknic
Used to view and set configure the VMkernel ports on virtual Ethernet switches. A VMkernel port is a special type of port group on a virtual Ethernet switch which is used to assign an IP address to the VMkernel. The VMkernel only needs an IP address for VMotion, software-initiated iSCSI or NFS access.
If you need to create a VMkernel port at the command line, then you need to create a port group first and then enable it as a VMkernel port. There doesn’t appear to be a way of enabling a VMkernel port for VMotion from the command line.
[root@esx1 root]# esxcfg-vswitch -A VMotion vSwitch0
[root@esx1 root]# esxcfg-vmknic -a -i 100.100.100.121 -n 255.255.255.0 VMotion
In the following example, we list the VMkernel ports, then delete one of them and then list them again.
[root@esx1 etc]# esxcfg-vmknic -l
Port Group IP Address Netmask Broadcast MAC Address MTU EnabledNFS access 100.100.100.21 255.255.255.0 100.100.100.255 00:50:56:62:ca:f6 1514 trueVMotion 100.100.100.121 255.255.255.0 100.100.100.255 00:50:56:6d:7c:7d 1514 true
[root@esx1 etc]# esxcfg-vmknic -d VMotion[root@esx1 etc]# esxcfg-vmknic -l
Port Group IP Address Netmask Broadcast MAC Address MTU EnabledNFS access 100.100.100.21 255.255.255.0 100.100.100.255 00:50:56:62:ca:f6 1514 true
esxcfg-vmknic [[]]
-a--add Add a VMkernel NIC to the system, requires IP parameters and portgroup name.
-d--del Delete VMkernel NIC on given portgroup.
-e--enable Enable the given NIC if disabled.
-D--disable Disable the given NIC if enabled.
-l--list List VMkernel NICs.
-i--ip The IP address for this VMkernel NIC. Setting an IP address requires that the --netmask option be given in same command.
-n--netmask The IP netmask for this VMkernel NIC. Setting the IP netmask requires that the --ip option be given in the same command.
-r--restore Restore VMkernel TCP/IP interfaces from Configuration file (FOR INTERNAL USE ONLY).
-h--help Show this message.
esxcfg-dumppart
Used to configure the VMkernel crash dump partition. The old ESX 2.x utility for this function (vmkdump) is still present on an ESX 3 server, but appears just to be for extracting dump files.
esxcfg-dumppart []
-l--list List the partitions available for Dump Partitions. WARNING: This will scan all LUNs on the system.
-t--get-active Get the active Dump Partition for this system, returns the internal name of the partition vmhbaX:X:X:X) or 'none'.
-c--get-config Get the configured Dump Partition for this system, returns the internal name of the partition vmhbaX:X:X:X) or 'none'.
-s--set Set the Dump Partition for this system and activate it, either vmhbaX:X:X:X or 'none' to deactivate the active dump partition.
-f--find Find usable Dump partitions and list in order of preference.
-S--smart-activate Activate the configured dump partition or find the first appropriate partition and use it (same order as -f).
-a--activate Activate the configured dump partition.
-d--deactivate Deactivate the active dump partition.
-h--help Show this message.
esxcfg-linuxnet
esxcfg-linuxnet
--setup
--remove
-h --help
The --setup option cannot be combined with the --remove option.
esxcfg-nics
This tool can be used to view and configure the speed and duplex settings of the physical network cards in the ESX Server. So this tool can replace the MUI Network Connections/Physical Adapters, the mii-tool and modules.conf for network card management,
In the following example, we run the list option to view all physical NICs and their properties.
[root@esx-v3 etc]# esxcfg-nics -l
Name PCI Driver Link Speed Duplex Descriptionvmnic2 01:01.00 tg3 Up 1000Mbps Full Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernetvmnic0 01:02.00 tg3 Up 100Mbps Full Broadcom Corporation NC7781 Gigabit Server Adapter (PCI-X, 10,100,1000-T)vmnic1 04:02.00 tg3 Up 1000Mbps Full Broadcom Corporation NC7781 Gigabit Server Adapter (PCI-X, 10,100,1000-T)
This command has the following optional parameters:
esxcfg-nics [nic]
-s--speed Set the speed of this NIC to one of 10/100/1000/10000. Requires a NIC parameter.
-d--duplex Set the duplex of this NIC to one of 'full' or 'half'. Requires a NIC parameter.
-a--auto Set speed and duplexity automatically. Requires a NIC parameter.
-l--list Print the list of NICs and their settings.
-r--restore Restore the nics configured speed/duplex settings (INTERNAL ONLY)
-h--help Display this message.
esxcfg-swiscsi
ESX version 3.0 supports both hardware and software iSCSI. For hardware iSCSI, we can use host bus adapters which perform the TCP offload and so the vmkernel can just pass SCSI commands to them as normal. The iSCSI hba can then wrap the SCSI command in TCP/IP and forward to the iSCSI target.
However, in software iSCSI (swiscsi), the wrapping of SCSI commands in TCP/IP is performed by the VMkernel and a regular physical network card can be used to communicate with the iSCSI target. This is exposed in the VI Client as a host bus adapter called vmhba40.
This will place a significant load on the VMkernel and wouldn't be that great an idea, but the feature is in ESX 3.0! So we use this tool esxcfg-swiscsi to configure it. The software iSCSI initiator in the VMkernel has a dependency upon the service console, therefore both the service console and VMkernel must have an IP route to the iSCSI target.
I have found that you need this command to scan for a new iSCSI target, as the VI Client rescan of the vmhba40 adapter doesn't appear to successfully discover targets.
My suggestion for getting the software iSCSI to work is as follows:
1. Add a VMkernel port to a vSwitch that has an uplink and route to iSCSI target
2. Ensure service console IP interface has a route to the same iSCSI target
3. Using either the VI Client security profile or the esxcfg-firewall, open a service console port for iSCSI (TCP:3260)
4. In the VI Client, enable the vmhab40 software iSCSI adapter and wait for the reconfiguration task to change from "In Progress" to "Completed"
5. Reboot the ESX host. This step will result in the VMkernel module for iSCSI being loaded at next boot.
6. In the VI Client, configure the vmhba40 adapter with an iSCSI target IP address
7. At the service console command line, run esxcfg-swiscsi -e
8. At the service console command line, run esxcfg-swiscsi -d
9. At the service console command line, run esxcfg-swiscsi -e
10. At the service console command line, run esxcfg-swiscsi -s
11. In the VI Client, perform a rescan of the vmhba adapters and your iSCSI target should become visible.
The command line options for this command are:
-e, --enable Enable sw iscsi
-d, --disable Disable sw iscsi
-q, --query Check if sw iscsi is on/off
-s, --scan Scan for disk available through sw iscsi interface
-k, --kill Try to forcibly remove iscsi sw stack
-r, --restore Restore sw iscsi configuration from file (FOR INTERNAL USE ONLY)
-h, --help Show this message
esxcfg-vswif
Manages the Ethernet interfaces of the service console
/etc/vmware/esx.conf
An all new configuration file for ESX Server 3.0. This file replaces the functionality of the following configuration files found in earlier versions of ESX.
/etc/vmware/hwconfig
/etc/vmware/devnames.conf
/etc/vmware/vmkmodule.conf
/etc/vmware/netmap.conf
/etc/vmware/vmkconfig
hostd
This is the daemon that replaces vmware-serverd. We can restart this with
service mgmt-vmware restart
vpxa
This is the name of the VirtualCenter server agent that runs in the service console of ESX 3.0 servers (was called vmware-ccagent in ESX 2.x). This can be stopped, started or restarted with the service command
service vmware-vpxa restart
/etc/vmware/vpxa.cfg
This is the XML configuration file for the VirtualCenter Server Agent in the service console. Here is a typical vpxa.cfg file.
[root@esx1 vmware]# cat vpxa.cfg
false
error
false
10
root
100.100.100.11 30 100.100.100.172
902
/var/log/vmware/vpx
vpxd
This is the process name of the Windows service that is the core service running on the VirtualCenter server.
vmkfstools
Used to manipulate virtual disks at the service console command line. It is used most often for import and export operations, where a virtual disk is converted from monolithic format to sparse format (previously called COW format).
There is a great switch with the command -X which can be used to extend the size of your virtual disk; e.g. if you had a 10GB virtual disk and wanted to expand it to 20GB, you could use this command. The VM would need to be powered off for this to work.
vmkfstools -X 20GB /vmfs/volumes/storage1/vm.vmdk
Note that the -X switch specifies the NEW SIZE of the virtual disk and NOT how much you are extending it by.
If you have used the -X switch before in an older version of ESX server (earlier than 3.0) it was possible to specify a small disk size; thereby making the virtual disk smaller. This was dangerous but useful if your partition within the disk did not consume 100% of the disk size. However, this is not possible with vmkfstools command found in ESX Server version 3.x
AAM
Automated Availability Manager that now runs in the service console when you create a VMware High Availability (VMware HA) cluster. The VMware HA feature was previously known as DAS (Distributed Availability Services) but we don't mention that anymore.
This software maintains an in-memory database on active nodes in the cluster and uses heartbeats to co-ordinate the active and passive nodes. It is suggested that you configure service console with 2 ethernet interfaces to remove any single point of failure.
This is a piece of licensed Legato software which itself has been renamed to EMC AutoStart.
This component has a very high dependency upon fully functional host name resolution. So before you enable VMware HA, check your /etc/hosts file, and your /etc/resolv.conf file to ensure accuracy. The log file for VMware HA can be found in the service console in the directory
/opt/LGTOaam512/
To avoid split brain scenarios, an ESX server can determine if it has become isolated from other servers and we can configure that servers' isolation response. If the AAM component loses contact with the other nodes in the HA cluster, it attempts to contact the configured default gateway for service console using ICMP echo request (PING). If this fails, then the ESX host is isolated. If your default gateway suppresses ICMP echo requests, then we can configure an alternate IP address called the das.isolationaddress.
This will briefly discuss on some commands new to ESX3.
esxcfg-
This is a new unified tool that is used to configure a large number of items in ESX Server 3.0.
esxcfg-advcfg
The esxcfg-advcfg command is interesting as there is not a huge amount of help about this command. However, we can figure out that it is meant to do advanced configuration and we can figure out some settings that can be made. The -g switch is used to "get" settings; the -s switch is used to "set" settings.
[root@esx1 vmware]# esxcfg-advcfg -g /Misc/BlueScreenTimeout
Value of BlueScreenTimeout is 0
[root@esx1 vmware]# esxcfg-advcfg -g /Misc/HostName
Value of HostName is esx1.vmlab.net
[root@esx1 vmware]# esxcfg-advcfg -g /VMFS3/ZeroedThickVirtualDisks
Value of ZeroedThickVirtualDisks is 1
[root@esx1 vmware]# esxcfg-advcfg –g /Disk/SupportSparseLUN
Value of SupportSparseLUN is 1
The question is, how much is configurable? To figure out what is configurable, we recommend that you look in the directory /proc/vmware/config in the service console and you will see the directories :
BufferCache
Cpu
Disk
FileSystem
Irq
LVM
Mem
Migrate
Misc
Net
NFS
Numa
Scsi
User
VMFS3
From these directories and the files within, you can work out the paths to be supplied to the esxcfg-advcfg command as parameters. Remember case sensitivity.
Usage: esxcfg-advcfg
-g--get Get the value of the config option
-s--set
-d--default Reset Config option to default
-q--quiet Suppress output
-k--set-kernel Set a VMkernel load time option value.
-j--get-kernel Get a VMkernel load time option value.
-h--help Show this message.
-r--restore Restore all advanced options from the configuration file. (FOR INTERNAL USE ONLY).
esxcfg-firewall
The service console in ESX 3 now has a firewall enabled by default. We use this command to view and configure the firewall rules.
The most popular switch will be the -q switch to query the firewall for its settings.
[root@esxhost1 root]# esxcfg-firewall -q
The -s switch will allow you to enable or disable network services that may traverse the firewall successfully. The list of known services are shown below - very case sensitive!....
nfsClient
ftpServer
ntpClient
dellom
nisClient
vncServer
tmpLicenseClient
swISCSIClient
CIMHttpsServer
sshClient
snmpd
tmpAAMClient
vpxHeartbeats
smbClient
hpim
tmpHostVmdbServer
tmpHostdSOAPServer
ftpClient
sshServer
ibmdirector
CIMHttpServer
telnetClient
The -q switch queries the state of a particular known service.
The -l switch loads the firewall and enables the IP tables.
The -u switch unloads the firewall and disables the IP tables.
We use the -e switch to enable a particular known service, so if we wanted to enable ssh outbound connections from the service console we would simply enter
[root@esxhost1 root]# esxcfg-firewall -e sshClient
We use the -d switch to disable a service. In the following example, we prevent outbound connections
[root@esxhost1 root]# esxcfg-firewall -d smbClient
esxcfg-module
This command produces an output similar to vmkload_mod -list
[root@lithium06 tools-isoimages]# esxcfg-module -l
Module Type Enabled Loaded
vmkapimod vmkapimod true true
vmklinux linux true true
cciss.o scsi true false
tg3.o nic true false
qla2300_7xx.o fc true false
Although if you compare the output with the old command, things don't exactly match up. Not sure why just yet....
esxcfg-rescan
As vmkfstools -rescan
esxcfg-upgrade
esxcfg-upgrade -h --help
-g --convert-grub
-f --convert-fstab
-r --upgrade-pre-vmkernel
-o --upgrade-post-vmkernel
The -g option may only be used with the -r option.
esxcfg-vswitch
This command allows you to list, add, modify or delete virtual Ethernet switches on an ESX host. The simplest option with this command is the -l option to list the virtual switches defined on the host.
[root@esx root]# esxcfg-vswitch -l
If you are having problems with your ESX server after an in-place upgrade, this tool is invaluable in resolving the problems with service console networking.
The output of this command is initially a little intimidating. It is best to keep in mind the network topology:
Service Console IP Interface (vswif0) ---- connected to ----> Service Console Port on vSwitch
----- uplinked to ----> vmnic
Where a vmnic is a physical Ethernet adapter in the ESX server.
To add a virtual Ethernet switch, we use "-a" with the command
[root@esx root]# esxcfg-vswitch -a vSwitch3
esxcfg-auth
Configures the service console authentication options including NIS, LDAP, Kerberos and Active Directory.
esxcfg-info
Produces an enormous amount of information about the ESX host. You really need to pipe this to a file for closer examination!
[root@esx root]# esxcfg-info >esxinfo.txt
esxcfg-mpath
Manages multi-pathing just as the vmkmultipath utility did in previous versions of ESX Server.
[root@lithium06 tools-isoimages]# esxcfg-mpath -l
Disk vmhba0:0:0 /dev/cciss/c0d0 (69459MB) has 1 paths and policy of Fixed
Local 2:1.0 vmhba0:0:0 On active preferred
Disk vmhba1:0:0 (0MB) has 1 paths and policy of Most Recently Used
FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:0
On active preferred
Disk vmhba1:0:6 /dev/sda (9216MB) has 1 paths and policy of Most Recently Used
FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:6
On active preferred
Disk vmhba1:0:21 /dev/sdb (10240MB) has 1 paths and policy of Most Recently Used
FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:21
On active preferred
esxcfg-resgrp
Used to manage the new ESX feature called resource groups. This command can add, remove or modify existing resource groups.
esxcfg-hbadevs
The esxcfg-vmhbadevs command is used to list the equivalent Linux device names for the visible disk devices that the VMkernel references using vmhba notation.
[root@esx1 root]# esxcfg-vmhbadevs
vmhba0:0:0 /dev/sda vmhba0:0:1 /dev/sdb
vmhba0:0:2 /dev/sdcvmhba0:0:3 /dev/sdd
vmhba2:0:0 /dev/sdevmhba2:1:0 /dev/sdf
If we use this command with the –m switch, then we only list the LUNs which contain VMFS partitions. Alongside the Linux device name, a long unique hexadecimal value is listed. This is the VMFS volume signature assigned by the new logical volume manager (LVM).
[root@esx1 root]# esxcfg-vmhbadevs -m
vmhba0:0:0:1 /dev/sda1 45407607-fbc43ced-94cb-00145e231ce3
vmhba0:0:2:1 /dev/sdc1 455b08a8-8af7fee3-daa9-00145e231e35
vmhba2:0:0:3 /dev/sde3 4559c75f-831d8f3e-bc81-00145e231e35
You can view these volumes in the directory /vmfs/volumes/
esxcfg-boot
Used to configure the GRUB options presented at boot time. One thing to note is that the new esxcfg commands will not run if you boot just into Linux. If you just want to query the boot settings, you can use the -q switch but this must be qualified with the keyword boot or vmkmod.
[root@lithium06 tools-isoimages]# esxcfg-boot -q boot
272 2:;7:;10:; UUID=847199e4-d3c7-11da-8ef8-930e3d734c03 /vmlinuz-2.4.21-37.0.2.ELvmnix /initrd-2.4.21-37.0.2.ELvmnix.img
[root@lithium06 tools-isoimages]# esxcfg-boot -q vmkmod
vmkapimod vmkapimod
vmklinux linux
cciss.o scsi
tg3.o nic
qla2300_7xx.o fc
Not yet tested other options....
esxcfg-init
Should not be run manually!
esxcfg-nas
Used to configure access to Network Attached Storage (NAS).
esxcfg-nas
-a--add Add a new NAS filesystem to /vmfs volumes. Requires --host and --share options.
-o--host
-s--share
-d--delete Unmount and delete a filesystem.
-l--list List the currently mounted NAS file systems.
-r--restore Restore all NAS mounts from the configuration file. (FOR INTERNAL USE ONLY).
-h--help Show this message.
esxcfg-route
If we add an IP address to the VMkernel by adding a VMkernel port, then we can fully configure that IP stack by also assigning a default gateway. We can view (no parameters) and set (1st parameter) the VMkernel IP default gateway with the esxcfg-route command as shown here.
[root@esx1 etc]# esxcfg-route
VMkernel default gateway is 100.100.100.254
[root@esx1 etc]# esxcfg-route 100.100.100.1
VMkernel default gateway set to 100.100.100.1
esxcfg-vmknic
Used to view and set configure the VMkernel ports on virtual Ethernet switches. A VMkernel port is a special type of port group on a virtual Ethernet switch which is used to assign an IP address to the VMkernel. The VMkernel only needs an IP address for VMotion, software-initiated iSCSI or NFS access.
If you need to create a VMkernel port at the command line, then you need to create a port group first and then enable it as a VMkernel port. There doesn’t appear to be a way of enabling a VMkernel port for VMotion from the command line.
[root@esx1 root]# esxcfg-vswitch -A VMotion vSwitch0
[root@esx1 root]# esxcfg-vmknic -a -i 100.100.100.121 -n 255.255.255.0 VMotion
In the following example, we list the VMkernel ports, then delete one of them and then list them again.
[root@esx1 etc]# esxcfg-vmknic -l
Port Group IP Address Netmask Broadcast MAC Address MTU EnabledNFS access 100.100.100.21 255.255.255.0 100.100.100.255 00:50:56:62:ca:f6 1514 trueVMotion 100.100.100.121 255.255.255.0 100.100.100.255 00:50:56:6d:7c:7d 1514 true
[root@esx1 etc]# esxcfg-vmknic -d VMotion[root@esx1 etc]# esxcfg-vmknic -l
Port Group IP Address Netmask Broadcast MAC Address MTU EnabledNFS access 100.100.100.21 255.255.255.0 100.100.100.255 00:50:56:62:ca:f6 1514 true
esxcfg-vmknic
-a--add Add a VMkernel NIC to the system, requires IP parameters and portgroup name.
-d--del Delete VMkernel NIC on given portgroup.
-e--enable Enable the given NIC if disabled.
-D--disable Disable the given NIC if enabled.
-l--list List VMkernel NICs.
-i--ip
-n--netmask
-r--restore Restore VMkernel TCP/IP interfaces from Configuration file (FOR INTERNAL USE ONLY).
-h--help Show this message.
esxcfg-dumppart
Used to configure the VMkernel crash dump partition. The old ESX 2.x utility for this function (vmkdump) is still present on an ESX 3 server, but appears just to be for extracting dump files.
esxcfg-dumppart
-l--list List the partitions available for Dump Partitions. WARNING: This will scan all LUNs on the system.
-t--get-active Get the active Dump Partition for this system, returns the internal name of the partition vmhbaX:X:X:X) or 'none'.
-c--get-config Get the configured Dump Partition for this system, returns the internal name of the partition vmhbaX:X:X:X) or 'none'.
-s--set Set the Dump Partition for this system and activate it, either vmhbaX:X:X:X or 'none' to deactivate the active dump partition.
-f--find Find usable Dump partitions and list in order of preference.
-S--smart-activate Activate the configured dump partition or find the first appropriate partition and use it (same order as -f).
-a--activate Activate the configured dump partition.
-d--deactivate Deactivate the active dump partition.
-h--help Show this message.
esxcfg-linuxnet
esxcfg-linuxnet
--setup
--remove
-h --help
The --setup option cannot be combined with the --remove option.
esxcfg-nics
This tool can be used to view and configure the speed and duplex settings of the physical network cards in the ESX Server. So this tool can replace the MUI Network Connections/Physical Adapters, the mii-tool and modules.conf for network card management,
In the following example, we run the list option to view all physical NICs and their properties.
[root@esx-v3 etc]# esxcfg-nics -l
Name PCI Driver Link Speed Duplex Descriptionvmnic2 01:01.00 tg3 Up 1000Mbps Full Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernetvmnic0 01:02.00 tg3 Up 100Mbps Full Broadcom Corporation NC7781 Gigabit Server Adapter (PCI-X, 10,100,1000-T)vmnic1 04:02.00 tg3 Up 1000Mbps Full Broadcom Corporation NC7781 Gigabit Server Adapter (PCI-X, 10,100,1000-T)
This command has the following optional parameters:
esxcfg-nics
-s--speed
-d--duplex
-a--auto Set speed and duplexity automatically. Requires a NIC parameter.
-l--list Print the list of NICs and their settings.
-r--restore Restore the nics configured speed/duplex settings (INTERNAL ONLY)
-h--help Display this message.
esxcfg-swiscsi
ESX version 3.0 supports both hardware and software iSCSI. For hardware iSCSI, we can use host bus adapters which perform the TCP offload and so the vmkernel can just pass SCSI commands to them as normal. The iSCSI hba can then wrap the SCSI command in TCP/IP and forward to the iSCSI target.
However, in software iSCSI (swiscsi), the wrapping of SCSI commands in TCP/IP is performed by the VMkernel and a regular physical network card can be used to communicate with the iSCSI target. This is exposed in the VI Client as a host bus adapter called vmhba40.
This will place a significant load on the VMkernel and wouldn't be that great an idea, but the feature is in ESX 3.0! So we use this tool esxcfg-swiscsi to configure it. The software iSCSI initiator in the VMkernel has a dependency upon the service console, therefore both the service console and VMkernel must have an IP route to the iSCSI target.
I have found that you need this command to scan for a new iSCSI target, as the VI Client rescan of the vmhba40 adapter doesn't appear to successfully discover targets.
My suggestion for getting the software iSCSI to work is as follows:
1. Add a VMkernel port to a vSwitch that has an uplink and route to iSCSI target
2. Ensure service console IP interface has a route to the same iSCSI target
3. Using either the VI Client security profile or the esxcfg-firewall, open a service console port for iSCSI (TCP:3260)
4. In the VI Client, enable the vmhab40 software iSCSI adapter and wait for the reconfiguration task to change from "In Progress" to "Completed"
5. Reboot the ESX host. This step will result in the VMkernel module for iSCSI being loaded at next boot.
6. In the VI Client, configure the vmhba40 adapter with an iSCSI target IP address
7. At the service console command line, run esxcfg-swiscsi -e
8. At the service console command line, run esxcfg-swiscsi -d
9. At the service console command line, run esxcfg-swiscsi -e
10. At the service console command line, run esxcfg-swiscsi -s
11. In the VI Client, perform a rescan of the vmhba adapters and your iSCSI target should become visible.
The command line options for this command are:
-e, --enable Enable sw iscsi
-d, --disable Disable sw iscsi
-q, --query Check if sw iscsi is on/off
-s, --scan Scan for disk available through sw iscsi interface
-k, --kill Try to forcibly remove iscsi sw stack
-r, --restore Restore sw iscsi configuration from file (FOR INTERNAL USE ONLY)
-h, --help Show this message
esxcfg-vswif
Manages the Ethernet interfaces of the service console
/etc/vmware/esx.conf
An all new configuration file for ESX Server 3.0. This file replaces the functionality of the following configuration files found in earlier versions of ESX.
/etc/vmware/hwconfig
/etc/vmware/devnames.conf
/etc/vmware/vmkmodule.conf
/etc/vmware/netmap.conf
/etc/vmware/vmkconfig
hostd
This is the daemon that replaces vmware-serverd. We can restart this with
service mgmt-vmware restart
vpxa
This is the name of the VirtualCenter server agent that runs in the service console of ESX 3.0 servers (was called vmware-ccagent in ESX 2.x). This can be stopped, started or restarted with the service command
service vmware-vpxa restart
/etc/vmware/vpxa.cfg
This is the XML configuration file for the VirtualCenter Server Agent in the service console. Here is a typical vpxa.cfg file.
[root@esx1 vmware]# cat vpxa.cfg
vpxd
This is the process name of the Windows service that is the core service running on the VirtualCenter server.
vmkfstools
Used to manipulate virtual disks at the service console command line. It is used most often for import and export operations, where a virtual disk is converted from monolithic format to sparse format (previously called COW format).
There is a great switch with the command -X which can be used to extend the size of your virtual disk; e.g. if you had a 10GB virtual disk and wanted to expand it to 20GB, you could use this command. The VM would need to be powered off for this to work.
vmkfstools -X 20GB /vmfs/volumes/storage1/vm.vmdk
Note that the -X switch specifies the NEW SIZE of the virtual disk and NOT how much you are extending it by.
If you have used the -X switch before in an older version of ESX server (earlier than 3.0) it was possible to specify a small disk size; thereby making the virtual disk smaller. This was dangerous but useful if your partition within the disk did not consume 100% of the disk size. However, this is not possible with vmkfstools command found in ESX Server version 3.x
AAM
Automated Availability Manager that now runs in the service console when you create a VMware High Availability (VMware HA) cluster. The VMware HA feature was previously known as DAS (Distributed Availability Services) but we don't mention that anymore.
This software maintains an in-memory database on active nodes in the cluster and uses heartbeats to co-ordinate the active and passive nodes. It is suggested that you configure service console with 2 ethernet interfaces to remove any single point of failure.
This is a piece of licensed Legato software which itself has been renamed to EMC AutoStart.
This component has a very high dependency upon fully functional host name resolution. So before you enable VMware HA, check your /etc/hosts file, and your /etc/resolv.conf file to ensure accuracy. The log file for VMware HA can be found in the service console in the directory
/opt/LGTOaam512/
To avoid split brain scenarios, an ESX server can determine if it has become isolated from other servers and we can configure that servers' isolation response. If the AAM component loses contact with the other nodes in the HA cluster, it attempts to contact the configured default gateway for service console using ICMP echo request (PING). If this fails, then the ESX host is isolated. If your default gateway suppresses ICMP echo requests, then we can configure an alternate IP address called the das.isolationaddress.
Friday, 15 June 2007
Syslog
Most network active equipment and some applications (mostly Linux ones) use the syslog utility to export all their errors and status messages to files located in the /var/log directory.
syslog is a utility for tracking and logging system messages from levels informational to critical.
Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle.
The first describes the function (aka facility) of the application that generated it. For example, applications such as mail and cron generate messages with facilities named mail and cron.
The second describes the degree of severity of the message.
You can configure syslog's /etc/syslog.conf configuration file to place messages of differing severities and facilities in different files.
syslog is a utility for tracking and logging system messages from levels informational to critical.
Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle.
The first describes the function (aka facility) of the application that generated it. For example, applications such as mail and cron generate messages with facilities named mail and cron.
The second describes the degree of severity of the message.
You can configure syslog's /etc/syslog.conf configuration file to place messages of differing severities and facilities in different files.
Other keywords may also be emerg, alert, crit, err, warning, notice, info, debug, none.
The output to which syslog writes each type of message received is set in the /etc/syslog.conf configuration file. This is done via 2 fields: The first lists the facilities and severities of messages to expect and the second lists the files to which they should be logged.
Some operating systems/devices, by default, put most of their messages in the file /var/log/messages.
Example:
*.info;mail.none;authpriv.none;cron.none /var/log/messages
(edit and post about vmbk, scripts to create VMs, cron, nfs, esx3 (nfs, ftp, firewall)
To test the syslog server, issue the following command to force a specific message to be sent:
logger -p local1.warning "SYSLOG TEST #1"
or another example:
logger -p local0.crit "Hello readers..."
__________________________________
In this case, all messages of severity "info" and above are logged, but none from the mail, cron or authentication facilities/subsystems. You can make this logging even more sensitive by replacing the line above with one that captures all messages from debug severity and above in the /var/log/messages file. This example may be more suitable for troubleshooting. *.debug /var/log/messages
In this example, all debug severity messages; except auth, authpriv, news and mail; are logged to the /var/log/debug file in caching mode. Notice how you can spread the configuration syntax across several lines using the slash (\) symbol at the end of each line. *.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
Here we see the /var/log/messages file configured in caching mode to receive only info, notice and warning messages except for the auth, authpriv, news and mail facilities. *.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
You can even have certain types of messages sent to the screen of all logged in users. In this example messages of severity emergency and above triggers this type of notification. The file definition is simply replaced by an asterisk to make this occur. *.emerg *
Certain applications will additionally log to their own application specific log files and directories independent of the syslog.conf file. Here are some common examples:
Files: /var/log/maillog : Mail
/var/log/httpd/access_log : Apache web server page access logs
Directories: /var/log
/var/log/samba : Samba messages
/var/log/mrtg : MRTG messages
/var/log/httpd : Apache webserver messages
Note: In some older versions of Linux the /etc/syslog.conf file was very sensitive to spaces and would recognize only tabs. The use of spaces in the file would cause unpredictable results. Check the formatting of your /etc/syslog.conf file to be safe.
Activating Changes to the syslog Configuration File
Changes to /etc/syslog.conf will not take effect until you restart syslog. Issue this command to do so: [root@bigboy tmp]# service syslog restart
In Ubuntu / Debian systems the command would be: root@u-bigboy:~# /etc/init.d/sysklogd restart
How to View New Log Entries as They Happen
If you want to get new log entries to scroll on the screen as they occur, then you can use this command: [root@bigboy tmp]# tail -f /var/log/messages
Similar commands can be applied to all log files. This is probably one of the best troubleshooting tools available in Linux. Another good command to use apart from tail is grep. grep will help you search for all occurrences of a string in a log file; you can pipe it through the more command so that you only get one screen at a time. Here is an example: [root@bigboy tmp]# grep string /var/log/messages more
You can also just use the plain old more command to see one screen at a time of the entire log file without filtering with grep. Here is an example: [root@bigboy tmp]# more /var/log/messages
Logging syslog Messages to a Remote Linux Server
Logging your system messages to a remote server is a good security practice. With all servers logging to a central syslog server, it becomes easier to correlate events across your company. It also makes covering up mistakes or malicious activities harder because the purposeful deletion of log files on a server cannot simultaneously occur on your logging server, especially if you restrict the user access to the logging server.
Configuring the Linux Syslog Server
By default syslog doesn't expect to receive messages from remote clients. Here's how to configure your Linux server to start listening for these messages.
As we saw previously, syslog checks its /etc/syslog.conf file to determine the expected names and locations of the log files it should create. It also checks the file /etc/sysconfig/syslog to determine the various modes in which it should operate. Syslog will not listen for remote messages unless the SYSLOGD_OPTIONS variable in this file has a -r included in it as shown below. # Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages received with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-m 0 -r"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
# once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
KLOGD_OPTIONS="-2"
Note: In Debian / Ubuntu systems you have to edit the syslog startup script /etc/init.d/sysklogd directly and make the SYSLOGD variable definition become "-r". # Options for start/restart the daemons
# For remote UDP logging use SYSLOGD="-r"
#
#SYSLOGD="-u syslog"
SYSLOGD="-r"
You will have to restart syslog on the server for the changes to take effect. The server will now start to listen on UDP port 514, which you can verify using either one of the following netstat command variations. [root@bigboy tmp]# netstat -a grep syslog
udp 0 0 *:syslog *:*
[root@bigboy tmp]# netstat -an grep 514
udp 0 0 0.0.0.0:514 0.0.0.0:*
[root@bigboy tmp]#
Configuring the Linux Client
The syslog server is now expecting to receive syslog messages. You have to configure your remote Linux client to send messages to it. This is done by editing the /etc/hosts file on the Linux client named smallfry. Here are the steps:
1) Determine the IP address and fully qualified hostname of your remote logging host.
2) Add an entry in the /etc/hosts file in the format: IP-address fully-qualified-domain-name hostname "loghost"
Example: 192.168.1.100 bigboy.my-site.com bigboy loghost
Now your /etc/hosts file has a nickname of "loghost" for server bigboy.
3) The next thing you need to do is edit your /etc/syslog.conf file to make the syslog messages get sent to your new loghost nickname. *.debug @loghost
*.debug /var/log/messages
You have now configured all debug messages and higher to be logged to both server bigboy ("loghost") and the local file /var/log/messages. Remember to restart syslog to get the remote logging started.
You can now test to make sure that the syslog server is receiving the messages with a simple test such as restarting the lpd printer daemon and making sure the remote server sees the messages.
Linux Client [root@smallfry tmp]# service lpd restart
Stopping lpd: [ OK ]
Starting lpd: [ OK ]
[root@smallfry tmp]#
Linux Server [root@bigboy tmp]# tail /var/log/messages
...
...
Apr 11 22:09:35 smallfry lpd: lpd shutdown succeeded
Apr 11 22:09:39 smallfry lpd: lpd startup succeeded
...
...
[root@bigboy tmp]#
Syslog Configuration and Cisco Network Devices
syslog reserves facilities "local0" through "local7" for log messages received from remote servers and network devices. Routers, switches, firewalls and load balancers each logging with a different facility can each have their own log files for easy troubleshooting. Appendix 4 has examples of how to configure syslog to do this with Cisco devices using separate log files for the routers, switches, PIX firewalls, CSS load balancers and LocalDirectors.
Logrotate
The Linux utility logrotate renames and reuses system error log files on a periodic basis so that they don't occupy excessive disk space.
The /etc/logrotate.conf File
This is logrotate's general configuration file in which you can specify the frequency with which the files are reused.
You can specify either a weekly or daily rotation parameter. In the case below the weekly option is commented out with a #, allowing for daily updates.
The rotate parameter specifies the number of copies of log files logrotate will maintain. In the case below the 4 copy option is commented out with a #, while allowing 7 copies.
The create parameter creates a new log file after each rotation
Therefore, our sample configuration file will create daily archives of all the logfiles and store them for seven days. The files will have the following names with, logfile being current active version: logfile
logfile.0
logfile.1
logfile.2
logfile.3
logfile.4
logfile.5
logfile.6
Sample Contents of /etc/logrotate.conf # rotate log files weekly
#weekly
# rotate log files daily
daily
# keep 4 weeks worth of backlogs
#rotate 4
# keep 7 days worth of backlogs
rotate 7
# create new (empty) log files after rotating old ones
create
The /etc/logrotate.d Directory
Most Linux applications that use syslog will put an additional configuration file in this directory to specify the names of the log files to be rotated. It is a good practice to verify that all new applications that you want to use the syslog log have configuration files in this directory. Here are some sample files that define the specific files to be rotated for each application.
Here is an example of a custom file located in this directory that rotates files with the .tgz extension which are located in the /data/backups directory. The parameters in this file will override the global defaults in the /etc/logrotate.conf file. In this case, the rotated files won't be compressed, they'll be held for 30 days only if they are not empty, and they will be given file permissions of 600 for user root. /data/backups/*.tgz {
daily
rotate 30
nocompress
missingok
notifempty
create 0600 root root
}
Note: In Debian / Ubuntu systems the /etc/cron.daily/sysklogd script reads the /etc/syslog.conf file and rotates any log files it finds configured there. This eliminates the need to create log rotation configuration files for the common system log files in the /etc/logrotate.d directory. As the script resides in the /etc/cron.daily directory it automatically runs every 24 hours. In Fedora / Redhat systems this script is replaced by the /etc/cron.daily/logrotate daily script which does not use the contents of the syslog configuration file, relying mostly on the contents of the /etc/logrotate.d directory.
Activating logrotate
The above logrotate settings in the previous section will not take effect until you issue the following command: [root@bigboy tmp]# logrotate -f
If you want logrotate to reload only a specific configuration file, and not all of them, then issue the logrotate command with just that filename as the argument like this: [root@bigboy tmp]# logrotate -f /etc/logrotate.d/syslog
Compressing Your Log Files
On busy Web sites the size of your log files can become quite large. Compression can be activated by editing the logrotate.conf file and adding the compress option. #
# File: /etc/logrotate.conf
#
# Activate log compression
compress
The log files will then start to become archived with the gzip utility, each file having a .gz extension. [root@bigboy tmp]# ls /var/log/messages*
/var/log/messages /var/log/messages.1.gz /var/log/messages.2.gz
/var/log/messages.3.gz /var/log/messages.4.gz /var/log/messages.5.gz
/var/log/messages.6.gz /var/log/messages.7.gz
[root@bigboy tmp]#
Viewing the contents of the files still remains easy because the zcat command can quickly output their contents to the screen. Use the command with the compressed file's name as the argument as seen below. [root@bigboy tmp]# zcat /var/log/messages.1.gz
...
...
Nov 15 04:08:02 bigboy httpd: httpd shutdown succeeded
Nov 15 04:08:04 bigboy httpd: httpd startup succeeded
Nov 15 04:08:05 bigboy sendmail[6003]: iACFMLHZ023165: to=
[root@bigboy tmp]#
syslog-ng
The more recent syslog-ng application combines the features of logrotate and syslog to create a much more customizable and feature rich product. This can be easily seen in the discussion of its configuration file that follows.
The /etc/syslog-ng/syslog-ng.conf file
The main configuration file for syslog-ng is the /etc/syslog-ng/sylog-ng.conf file but only rudimentary help on its keywords can be found using the Linux man pages. [root@zippy tmp]# man syslog-ng.conf
Figure 5-1 has a sample syslog-ng.conf file and outlines some key features. The options section that covers global characteristics is fully commented, but it is the source, destination and log sections that define the true strength of the customizability of syslog-ng.
Figure 5-1 A Sample syslog-ng.conf File options {
# Number of syslog lines stored in memory before being written to files
sync (0);
# Syslog-ng uses queues
log_fifo_size (1000);
# Create log directories as needed
create_dirs (yes);
# Make the group "logs" own the log files and directories
group (logs);
dir_group (logs);
# Set the file and directory permissions
perm (0640);
dir_perm (0750);
# Check client hostnames for valid DNS characters
check_hostname (yes);
# Specify whether to trust hostname in the log message.
# If "yes", then it is left unchanged, if "no" the server replaces
# it with client's DNS lookup value.
keep_hostname (yes);
# Use DNS fully qualified domain names (FQDN)
# for the names of log file folders
use_fqdn (yes);
use_dns (yes);
# Cache DNS entries for up to 1000 hosts for 12 hours
dns_cache (yes);
dns_cache_size (1000);
dns_cache_expire (43200);
};
# Define all the sources of localhost generated syslog
# messages and label it "d_localhost"
source s_localhost {
pipe ("/proc/kmsg" log_prefix("kernel: "));
unix-stream ("/dev/log");
internal();
};
# Define all the sources of network generated syslog
# messages and label it "d_network"
source s_network {
tcp(max-connections(5000));
udp();
};
# Define the destination "d_localhost" log directory
destination d_localhost {
file ("/var/log/syslog-ng/$YEAR.$MONTH.$DAY/localhost/$FACILITY.log");
};
# Define the destination "d_network" log directory
destination d_network {
file ("/var/log/syslog-ng/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log");
};
# Any logs that match the "s_localhost" source should be logged
# in the "d_localhost" directory
log { source(s_localhost);
destination(d_localhost);
};
# Any logs that match the "s_network" source should be logged
# in the "d_network" directory
log { source(s_network);
destination(d_network);
};
In our example, the first set of sources is labeled s_localhost. It includes all system messages sent to the Linux /dev/log device, which is one of syslog's data sources, all messages that syslog-ng views as being of an internal nature and additionally inserts the prefix "kernel" to all messages it intercepts on their way to the /proc/kmsg kernel message file.
Unlike a regular syslog server which listens for client messages on UDP port 514, syslog-ng also listens on TCP port 514. The second set of sources is labeled s_network and includes all syslog messages obtained from UDP sources and limits TCP syslog connections to 5000. Limiting the number of connections to help regulate system load is a good practice in the event that some syslog client begins to inundate your server with messages.
Our example also has two destinations for syslog messages, one named d_localhost, the other, d_network. These examples show the flexibility of syslog-ng in using variables. The $YEAR, $MONTH and $DAY variables map to the current year, month and day in YYYY, MM and DD format respectively. Therefore the example: /var/log/syslog-ng/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log
refers to a directory called /var/log/syslog-ng/2005.07.09 when messages arrive on July 9, 2005. The $HOST variable refers to the hostname of the syslog client and will map to the client's IP address if DNS services are deactivated in the options section of the syslog-ng.conf file. Similarly the $FACILITY variable refers to the facility of the syslog messages that arrive from that host.
Installing syslog-ng
The most recent syslog-ng and its companion eventlog tar files can be downloaded from the www.balabit.com website. The installation procedure is straightforward, but you will need to have the Linux gcc C programming language compiler preinstalled to be successful. Here are the steps.
1. Download the tar files from the BalaBit website. In this case we have browsed the website beforehand and know the exact URLs to use with the wget command. [root@zippy tmp]# wget wget http://www.balabit.com/downloads/syslog-ng/2.0/src/eventlog-0.2.5.tar.gz
--12:34:17-- wget http://www.balabit.com/downloads/syslog-ng/2.0/src/eventlog-0.2.5.tar.gz
=> `eventlog-0.2.5.tar.gz'
...
...
...
12:34:19 (162.01 KB/s) - `eventlog-0.2.5.tar.gz' saved [345231]
[root@zippy tmp]# wget http://www.balabit.com/downloads/syslog-ng/2.0/src/syslog-ng-2.0.0.tar.gz
--12:24:21-- wget http://www.balabit.com/downloads/syslog-ng/2.0/src/syslog-ng-2.0.0.tar.gz
=> ` syslog-ng-2.0.0.tar.gz'
...
...
...
12:24:24 (156.15 KB/s) - ` syslog-ng-2.0.0.tar.gz' saved [383589]
[root@zippy tmp]#
2. Install the prerequisite glib libraries. [root@zippy tmp]# yum -y install glib
3. Using the tar command we extract the files in the pre-requisite eventlog archive and then use the configure; make and make install commands to install them correctly. Pay special attention to the output of the configure command to make sure that all the pre-installation tests are passed. If not, install the packages the error messages request and then start again. [root@zippy tmp]# tar -xzf eventlog-0.2.5.tar.gz
[root@zippy tmp]# cd eventlog-0.2.5
[root@zippy eventlog-0.2.5]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
...
...
[root@zippy eventlog-0.2.5]# make
Making all in utils
make[1]: Entering directory `/tmp/eventlog-0.2.5/utils'
sed -e "s,_SCSH_,/usr/bin/scsh," make_class.in >make_class
...
...
...
[root@zippy eventlog-0.2.5]# make install
Making install in utils
make[1]: Entering directory `/tmp/eventlog-0.2.5/utils'
make[2]: Entering directory `/tmp/eventlog-0.2.5/utils'
...
...
...
make[2]: Leaving directory `/tmp/eventlog-0.2.5'
make[1]: Leaving directory `/tmp/eventlog-0.2.5'
[root@zippy eventlog-0.2.5]#
4. The next step is to install the prerequisite glib package on your system. [root@zippy eventlog-0.2.5]# yum -y install glib
5. Some environmental variables also need to be set prior to the installation of the syslog-ng files. [root@zippy eventlog-0.2.5]# PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
[root@zippy eventlog-0.2.5]# export PKG_CONFIG_PATH
6. Using the tar command we extract the files in the pre-requisite syslog-ng archive and then use the configure, make clean, make and make install commands to install them correctly. In this case we the --sysconfdir directive with the configure command to make sure syslog-ng searches for its configuration file in the /etc directory. Once again, pay close attention to the pre-installation tests that the configure command executes. [root@zippy eventlog-0.2.5]# cd /tmp
[root@zippy tmp]# tar -xzf syslog-ng-2.0.0.tar.gz
[root@zippy tmp]# cd syslog-ng-2.0.0
[root@zippy syslog-ng-2.0.0]# make clean
[root@zippy syslog-ng-2.0.0]# ./configure --sysconfdir=/etc
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
...
...
[root@zippy syslog-ng-2.0.0]# make; make install
Making all in src
make[1]: Entering directory `/tmp/ syslog-ng-2.0.0/src'
...
...
...
[root@zippy syslog-ng-2.0.0]#
7. The installation has template init.d/syslog-ng scripts and syslog-ng.conf files in the contribs/ directory. [root@zippy syslog-ng-2.0.0]# ls contrib/
fedora-packaging init.d.RedHat-7.3 init.d.SuSE
Makefile.in rhel-packaging syslog-ng.conf.HP-UX
syslog-ng.vim init.d.HP-UX init.d.solaris
Makefile README syslog2ng
init.d.RedHat syslog-ng.conf.RedHat init.d.SunOS
Makefile.am relogger.pl syslog-ng.conf.doc
syslog-ng.conf.SunOS
[root@zippy syslog-ng-2.0.0]#
8. Copy the versions for your operating system to the /etc/init.d and /etc , /etc/logrotate.d , /etc/sysconfig directories. The /etc/syslog-ng/ directory needs to be created beforehand. Redhat and Fedora installations have their own subdirectories contrib/. [root@zippy syslog-ng-2.0.0]# mkdir /etc/syslog-ng/
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.init \
/etc/init.d/syslog-ng
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.conf \
/etc
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.sysconfig \
/etc/sysconfig/syslog-ng
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.logrotate \
/etc/logrotate.d/syslog-ng
Remember that you may want to customize your syslog-ng.conf file.
9. Change the permissions on your new /etc/inid.d/syslog-ng file. [root@zippy syslog-ng-2.0.0]# chmod 755 /etc/init.d/syslog-ng
10. You need to be careful. The init.d script may refer to a syslog-ng binary file that's in an incorrect location. Find its true location and edit the script. [root@zippy syslog-ng-2.0.0]# updatedb
[root@zippy syslog-ng-2.0.0]# locate syslog-ng grep bin
/usr/local/sbin/syslog-ng
[root@zippy syslog-ng-2.0.0]# vi /etc/init.d/syslog-ng
...
#exec="/sbin/syslog-ng"
exec="/usr/local/sbin/syslog-ng"
...
:wq
[root@zippy syslog-ng-2.0.0]#
11. Next create the /etc/syslog-ng directory for the configuration files and the /var/log/syslog-ng directory for the log files. [root@zippy syslog-ng-2.0.0]# chkconfig syslog off
[root@zippy syslog-ng-2.0.0]# chkconfig syslog-ng on
[root@zippy syslog-ng-2.0.0]# service syslog stop
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
[root@zippy syslog-ng-2.0.0]# service syslog-ng start
syslog-ng: unrecognized service
[root@zippy syslog-ng-2.0.0]#
12. The sample syslog-ng.conf file in Figure 5-1 was configured to have all directories owned by the group logs. This user group needs to be created and any users that need access to the directories need to added to this group using the usermod command. In this case the user peter is added to the group and the groups command is used to verify success. [root@zippy tmp]# groupadd logs
[root@zippy tmp]# usermod -G logs peter
[root@zippy tmp]# groups peter
peter: users logs
[root@zippy tmp]# usermod -G logs peter
13. You can now configure syslog-ng to start on the next reboot with the chkconfig command and then use the service command to start it immediately. Remember to stop the old syslog process beforehand. [root@zippy tmp]# service syslog stop
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
[root@zippy tmp]# chkconfig syslog off
[root@zippy tmp]# chkconfig syslog-ng on
[root@zippy tmp]# service syslog-ng start
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
[root@zippy tmp]#
14. Now, your remote hosts should log begin logging to the /var/log/syslog-ng directory. According to our preliminary configuration file, there should be sub-directories categorized by date inside it. Each of these sub-directories in turn will have directories beneath them named after the IP address and/or hostname of the various remote syslog clients and will contain files categorized by syslog facility. In this example we see that the 2005.07.09 directory as received messages from three hosts, 192.168.1.1, 192.168.1.100 and localhost. [root@zippy tmp]# ls /var/log/syslog-ng/
2005.07.09
[root@zippy tmp]# ll /var/log/syslog-ng/2005.07.09/
drwxr-x--- 2 root logs 4096 Jul 9 17:01 192-168-1-1.my-web-site.org
drwxr-x--- 2 root logs 4096 Jul 9 16:45 192-168-1-99.my-web-site.org
drwxr-x--- 2 root logs 4096 Jul 9 23:24 LOGGER
[root@zippy tmp]# ls /var/log/syslog-ng/2005.07.09/localhost/
cron.log kern.log local7.log syslog.log
[root@zippy tmp]#
Using syslog-ng your system can now be used as a much more customizable tool to help troubleshoot devices attached to your network. Each day syslog-ng will automatically create new sub-directories to match the current date and at the end of each calendar quarter the files will be moved to a special archive directory containing all the data for the previous three months. This archived data can then be periodically deleted as needed. For very large deployments, or for better searching and correlation capabilities, it is possible to send the output of syslog-ng to a SQL type database. This is beyond the scope of this book, but it is a worthwhile feature to keep in mind.
Configuring syslog-ng Clients
Clients logging to the syslog-ng server don't need to have syslog-ng installed on them, a regular syslog client configuration will suffice.
Simple syslog Security
One of the shortcomings of a syslog server is that it doesn't filter out messages from undesirable sources. It is therefore wise to implement the use of TCP wrappers or a firewall to limit the acceptable sources of messages when your server isn't located on a secure network. This will help to limit the effectiveness of syslog based denial of service attacks aimed at filling up your server's hard disk or taxing other system resources that could eventually cause the server to crash.
Remember that regular syslog servers listen on UDP port 514 and syslog-ng servers rely on port 514 for both UDP and TCP. Please refer to Chapter 14, "Linux Firewalls Using iptables", on Linux firewalls for details on how to configure the Linux iptables firewall application and Appendix I, "Miscellaneous Linux Topics", for further information on configuring TCP wrappers.
Conclusion
In the next chapter we cover the installation of Linux applications, and the use of syslog will become increasingly important especially in the troubleshooting of Linux-based firewalls which can be configured to ignore and then log all undesirable packets; the Apache Web server which logs all application programming errors generated by some of the popular scripting languages such as PERL and PHP; and finally, Linux mail whose configuration files are probably the most frequently edited system documents of all and which correspondingly suffer from the most mistakes.
This syslog chapter should make you more confident to learn more about these applications via experimentation because you'll at least know where to look at the first sign of trouble.
Some contents retrieved from "http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch05_:_Troubleshooting_Linux_with_syslog"
Some more useful links on syslog:
tutorials and guides
syslog home (tips and tools)
Practical Unix & Internet Security (O'Reilly)
Subscribe to:
Posts (Atom)