VMware Data Recovery

How do you backup your virtual infrastructure based on VMware platform? Do you still rely on VCB? Even if the VCB is still supported and it is compatible with the latest ESX/ESXi hypervisor at version 4.1u1 it should be discontinued from the next release. So what backup tool to use in the future? 

VMware vDR is considered to be its successor and in my opinion, the latest release is working pretty stable without any major issue. It's missing some features now like email reporting or GUI FLR restore tool but I think we can expect the future release will include them as the competing backup tools don't miss such basic things.

But what I consider the most important for better adoption of vDR in datacenters is that VMware corrected their licenses recently and vDR is now included in Standard edition as well. This means that vDR is available in all vSphere editions except vSphere Essentials kit.

DNS reverse mapping

Recently, I had to cope with configuring some reverse zones for subnets where the netmask is like 26 or 20 (IPv4). It's quite straightforward to do it with class C networks when it is sufficient to  reverse the order of network base of the address,  join the result with special domain in-addr.arpa and  create a reversed mapped zone file finally. If we had a network 192.168.1.0/24 then the reversed base of the network address would be 1.168.192 and the reversed zone 1.168.192.in-addr.arpa. 

The previous technique is well known for class C networks (B or A as well) when it is possible to split  it on its octet boundaries. If we have assigned a class C subnet which has less than 256 hosts we can't do it like this and we need to define the  network part and the host part of the address differently. The brute force way how to do it is to create a reversed zone for each host . The better way is to read through the RFC2317.

The RFC defines a classless allocation of subnets on non-octal boundaries with less than 256 hosts. Let's take a network 192.168.1.32/28 (subnet of network 192.168.1.0/24) where the network base is 192.168.1.32, the maximum number of host is 14 and the netmask is 255.255.255.240 (28 in CIDR notation). The next step is to reverse the network base which gives us 32.1.168.192 and join the result with the domain in-addr.arpa. It gives us a semi-reversed zone 32.1.168.192.in-addr.arpa. The final steps to construct the reversed zone are  not so clear. Take the first octet from the semi-reversed zone, substitute it with the netmask in CIDR notation and write it in the form first_octet/substituted_zone. In our example, we would get 32/28.1.168.192.in-addr.arpa (32 is the last octet and 28 is the netmask).

Why is the presented method useful? Even if the reverse zone creation is not so clear it helps to create only 1 zone file for 14 hosts in our example. If we had a network with netmask e.g. 25 we would be able to specify all 126 PTR records in one zone file. The method is obfuscated a bit but it eliminates  a creation of reversed zone file for each host.

It's important to realize the RFC2317 is for networks with 256 hosts or less  where  netmask  is from 24 to 32. For networks with more hosts there has to be used traditional delegation. If we have a network with netmask from 17 to 24 then we will have 1 zone file for each 256 hosts.  If we have a netmask from 9 to 16 then we can have 1 zone file per 65536 hosts.

Quickly - persistent modules loading on RHEL

The kernel modules required for booting the system up are part of an initial  ramdisk which is automatically loaded into the memory by a boot loader. The ramdisk contains enough modules to mount the root filesystem and to initialize essential devices like keyboard, console or   various expansion cards.  The boot process  then continues with running the init process.

During the next phase, the other modules referenced by the operating system  are loaded automatically.  The modules are called by their aliases specified and set in the /etc/modprobe.conf configuration file. The typical alias is e.g. eth0 for a network interface card or usb-controller for an USB controller.

If we need  to load some specific module during the system boot and there isn't a way to reference it we have a few choices how to do it.

• Place a particular modprobe command to the /etc/rc.d/rc.local script which is called at the end of the whole boot process. But it is likely to be late at this phase.
• Or better, place the command in the  /etc/rc.modules file which is read and executed by the /etc/rc.d/rc.sysinit initialization script during the system initialization phase. It may be better to load the modules as soon as possible.

The /etc/rc.modules does not exist by default, so at first create it and make it executable. I think the first method is commonly used by many of us but the second one is in my opinion more systematical.

ESXi log files

What is the fastest way to retrieve log files from an ESXi host? In my opinion, the best way is to configure remote logging via syslog server but this requires host reboot to apply configuration changes (KB1016621). The alternative method is to forward log files to different datastore. 

If  you don't have prepared syslog server for remote logging you can use vsphere client and generate system log bundles for particular host. But this takes some time. 

The last method is I think the fastest one because it will allow you to access log files  directly with your web browser. You can use web interface of the ESXi host,  enter the following URL:

https://ESXi_HOST_ADDR/host

The output should looks like shown at this picture:

You can download ESXi log files  messages, hostd.log and vpxa.log  now.

vMA missing libraries

If you are using vMA (vSphere Management Assistant) for some specific management tasks like UPS monitoring  or running a scheduled backup script from cron daemon, you may experience an error similar to this one:

Can't load '/usr/lib/perl5/site_perl/5.8.8/libvmatargetlib_perl.so'
for module vmatargetlib_perl: libtypes.so: cannot open shared object
file: No such file or directory at
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.
at /usr/lib/perl5/site_perl/5.8.8/VMware/VmaTargetLib.pm line 10
Compilation failed in require at /usr/lib/perl5/site_perl/5.8.8/VMware/VIFPLib.pm line 10.

A reason for such  behaviour is typically caused by some misunderstandings how shell environment in vMA is configured. The most common mistake is testing the affected script with sudo which strips out some environment variables - especially LD_LIBRARY_PATH - due to some security restrictions. Otherwise, the error shouldn't appear because /etc/bashrc exports vmware SDK library path implicitly:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib

So in  case of sudo or other unspecified scenarios throwing the presented error try to create a wrapper script which explicitly exports a list of directories where to search for ELF libraries again:

#!/bin/bash

LD_LIBRARY_PATH=/opt/vmware/vma/lib64:/opt/vmware/vma/lib  
export LD_LIBRARY_PATH

/path/to/original-script "$@"

exit $?

VCB basic usage - debugging

During the series of articles about VCB usage I supposed that all the presented VCB command examples are running smoothly and without errors. But this is not always true. There can  be a lot of reason why it is not working as expected, e.g. wrong permissions assigned to VCB backup user, misconfigured SAN which doesn't allow to access VMFS volumes or  unspecified problem with creating virtual machine snapshot.

If something goes wrong all VCB commands can be run in more verbose mode with command line switch -L and verbosity level from 0 to 6. The next example illustrates it. We want to perform a full backup of a virtual machine named vcb-backup and it seems the provided user vcbadmin doesn't have required permissions to do it:

vcbmounter -h host -u vcbadmin -p pass -a name:vcb-backup -r c:\mnt\vcb-backup -t fullvm -m nbd -L4

[2010-06-15 13:15:12.843 'vcbMounter' 360 info] Connected using API Namespace vim25.
[2010-06-15 13:15:12.843 'vcbMounter' 360 info] Authenticating user vcbadmin
[2010-06-15 13:15:12.859 'vcbMounter' 360 info] Logged in!
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got VM MoRef
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got access method
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Got coordinator object
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Attempting data access.
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Creating mount directory
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] No snapshot info for this VM, nothing to do.
[2010-06-15 13:15:12.890 'vcbMounter' 360 info] Creating snapshot
[2010-06-15 13:15:19.296 'vcbMounter' 360 info] Snapshot created, ID: snapshot-579
[2010-06-15 13:15:19.296 'vcbMounter' 360 info] Mount operation created snapshot.
[2010-06-15 13:15:19.312 'vcbMounter' 360 info] Mount operation obtained backupinfo.
[2010-06-15 13:15:19.312 'vcbMounter' 360 info] Performing SearchIndex find.
[2010-06-15 13:15:19.328 'vcbMounter' 360 info] Successfully obtained instance lock.

[2010-06-15 13:15:24.359 'vcbMounter' 360 error] Error: No permission to perform this action.

[2010-06-15 13:15:24.359 'vcbMounter' 360 error] An error occurred, cleaning up...
[2010-06-15 13:15:24.359 'vcbMounter' 360 info] Performing SearchIndex find.
[2010-06-15 13:15:24.359 'vcbMounter' 360 info] Successfully obtained instance lock.
[2010-06-15 13:15:29.437 'vcbMounter' 360 info] Remove clone disks successful.
Deleted directory c:\vcb\vcb-backup

The bold line helps us to identify the cause of the problem.

YUM download only mode

How many times I was in a situation I needed to update a server with RHEL installed but I wasn't at site and I didn't have a way how to reboot the server after installing a new kernel or glibc package on it reliably? Yes, I have a test environment and I'm testing the updates on it but many installations are too critical to just run yum update -y and then shutdown -r now. On top of that, there are well known Murphy's laws which are able to damage more than we are able to imagine.

Instead of remote resolution of why the server is suddenly unresponsive I'm trying to prepare some offline update archive (if there isn't an update server available but this is another situation) and then during a site visit to apply it.

As I'm talking about RHEL I'm using YUM or Yellowdog Updater Modified for it. This tool is able to download updates locally without installing them if we have RHEL 5.x system. It only requires to install a download plugin which is part of yum-downloadonly package. Try to install it with


yum install yum-downloadonly

The next lines contain common commands that I use for downloading updates:

yum install PACKAGE_NAME.rpm -y --downloadonly
yum update -y --downloadonly


If we have a RHEL 4.x server we don't have this package and we need to install another package called yum-tools which contains similar tool yumdownloader.


yum install yum-tools -y

Here it is how to use the tool

yumdownloader PACKAGE_NAME.rpm


If we wan't to download all the available updates with yumdownloader we need to get a list of all packages with yum check-update and then to pass it to yumdownloader. You can do it from shell with sed, cut or awk commands or what would you prefer:

for PKG in `yum check-update | cut -d' ' -f1`; do
   yumdownloader $PKG
done

For more detailed description of the tools and their parameters have a look at their man pages.

Time to continue

After almost a whole year, I would like to awake my blog from a sleep and begin to publish articles more regularly. During the previous period I was too busy at work and I couldn't afford to take care of it as I would like. On the other hand, I had an opportunity to participate on some interesting projects and I needed to solve many unusual tasks. Everything was written down and now it is waiting for readers and comments.

I hope the new articles will be interesting and helpful for you as the previous ones.

Red Hat Enterprise Linux 5.5 - what's new?

It's a few months since RHEL 5.5 was released (march, 2010). Despite this, I would like to point out the major changes and additions compared to the previous release RHEL 5.4. So what's new:

• Kickstart installation - it is possible to exclude package groups in the same way like single packages.
• KVM guests and Cluster Suite - management of KVM based virtual guests with Cluster Suite is supported.
• SPICE - RHEL 5.5 includes components of Simple Protocol for Independent Computing Environments which is competitor for VMware's PCoIP or Citrix's HDX.
• PCI passthrough - physical PCI devices attached to virtual guests are working better.
• Huge page support - it is extended to virtual guests with libvirt.
• Windows 7 support - new samba3x packages supporting Windows 7 are included.

For more details read the RHEL 5.5 official release notes.

VMware Workstation 7.1 enhancements

I have decided to update my current VMware workstation 7.0.1 installation I have on my desktop to the latest available release 7.1. Further follows a quick comparison of the most interesting features and enhancements between these two releases:

• WS7.1 officially support many latest versions of well-known Linux distros like RHEL/Debian/Ubuntu
• WS7.1 supports OVF1.1 via ovftool which is part of the installer
• WS7.1 has higher limits for guest's vCPUs - from 4 to 8 and guest's virtual disk - from 950GB to 2TB
• WS7.1 is capable of automatic downloads and updates of WMware Tools

Eventually, I would like to remember what new features VMware Workstation 7.0 came with:

• 32GB of guest's virtual RAM
• cross-platform license serial numbers
• pause running guest instead of suspending it or even powering it off
  
• import of Windows XP mode virtual machine
• GUI editor for virtual networks
• support for Windows 7 as guests
• support ESX hypervisor running as guest

That's not everything at all. The official release notes for VMware Workstation 7.1 and 7.0 are more comprehensive.